Prioritizing interventions and reproducibility measures to improve research reproducibility: a Delphi consultation method
Dora Pejdo
(Medical School, University of Split)
Ana Marušić
(University of Split School of Medicine)
Ivan Buljan
(Faculty of Humanities and Social Sciences in Split)
Short abstract
Under the Horizon Europe iRISE project, a Delphi consultation study mapped the priorities for reproducibility measures and interventions with experts across research domains. ”Methodological quality” scored highest as a reproducibility measure, and “Data management training” as an intervention.
Long abstract
Background
Horizon Europe iRISE project aims to improve research reproducibility. A Delphi consensus study was conducted to gather expert input across research domains and stakeholder groups: policymakers, funders, editors, publishers, and researchers, about the priorities in addressing reproducibility measures and interventions.
Methods
The study involved 73 experts from 34 countries in Consultation Round 1 and 67 in Round 2. Using a 10-point Likert scale, reproducibility measures (n=14) and interventions (n=27) required at least 70% of panellists scoring items 8 or higher to reach a consensus. The panellists had the option to comment on their choices. Comments from Round 1 were summarised using content analysis and provided to the panellists in Round 2. The final panel to create prioritized lists is in planning.
Results
From Round 1, a total of 8 reproducibility measures reached the priority list, along with 6 interventions. The top three priorities for reproducibility measures were “Methodological quality – 9.03”, “Reporting quality – 9.00”, and “Code and data availability and re-use – 8.66”. The top three priorities for interventions were “Data management training – 8.52”, Data quality checks/feedback – 8.33”, and “Statistical training – 8.33”. After Round 2 there were no new reproducibility measures or interventions that reached consensus to be included in the priority list.
Conclusions
This study identified actionable areas for improving reproducibility and transparency by consulting experts from diverse disciplines. The final panel will create the final priority list. Study findings will benefit the scientific community by shaping concrete actions to address critical reproducibility challenges.
Accepted Paper
Short abstract
Long abstract
Background
Horizon Europe iRISE project aims to improve research reproducibility. A Delphi consensus study was conducted to gather expert input across research domains and stakeholder groups: policymakers, funders, editors, publishers, and researchers, about the priorities in addressing reproducibility measures and interventions.
Methods
The study involved 73 experts from 34 countries in Consultation Round 1 and 67 in Round 2. Using a 10-point Likert scale, reproducibility measures (n=14) and interventions (n=27) required at least 70% of panellists scoring items 8 or higher to reach a consensus. The panellists had the option to comment on their choices. Comments from Round 1 were summarised using content analysis and provided to the panellists in Round 2. The final panel to create prioritized lists is in planning.
Results
From Round 1, a total of 8 reproducibility measures reached the priority list, along with 6 interventions. The top three priorities for reproducibility measures were “Methodological quality – 9.03”, “Reporting quality – 9.00”, and “Code and data availability and re-use – 8.66”. The top three priorities for interventions were “Data management training – 8.52”, Data quality checks/feedback – 8.33”, and “Statistical training – 8.33”. After Round 2 there were no new reproducibility measures or interventions that reached consensus to be included in the priority list.
Conclusions
This study identified actionable areas for improving reproducibility and transparency by consulting experts from diverse disciplines. The final panel will create the final priority list. Study findings will benefit the scientific community by shaping concrete actions to address critical reproducibility challenges.
Where next for replication, transparency and analysis of QRPs? (II)
Session 1 Tuesday 1 July, 2025, -