iRISE – SOLES: A Systematic Online Living Evidence Summary for interventions to improve reproducibility
Sarah Wendt
(Facultad Latinoamericana de Ciencias Sociales Argentina)
Dora Pejdo
(Medical School, University of Split)
Ivan Buljan
(Faculty of Humanities and Social Sciences in Split)
Ana Marušić
(University of Split School of Medicine)
Kimberley Wever
(Radboud university medical center)
Sarah McCann
(Berlin Institute of Health at Charité)
Sean Smith
(University of Edinburgh)
Torsten Rackoll
Maria Economou
Malcolm R. Macleod
(University of Edinburgh)
Kaitlyn Hair
(University College London)
Emily Sena
(University of Edinburgh)
Short abstract
To address the uncertainty around effective interventions for improving reproducibility in science, we developed SOLES - a Systematic Online Living Evidence Summary - using machine-assisted screening and AI-driven annotations, aided by community feedback, presented in an interactive web dashboard.
Long abstract
Reproducibility is fundamental to scientific progress. Multiple interventions to improve reproducibility have been proposed and/or tested, yet it remains unclear which strategies are most effective. As part of the iRISE (improving Reproducibility in SciencE) project, we have created a Systematic Online Living Evidence Summary (SOLES) to identify, curate, and visualise the entire literature base, aided by artificial intelligence (AI).
We systematically identified published articles describing interventions to improve reproducibility (n=16832 included). After dual-screening a subset (n=5000) for relevance, we trained a machine learning classifier to identify all relevant articles. We annotated 138 articles for predefined attributes, including scientific discipline, intervention, outcome, participants, and location. Using these annotations, we designed prompts and evaluated the annotation capabilities of different large-language models. The best performing approach was then applied across the included studies.
All outputs are presented on an interactive web dashboard, where users can interrogate the latest evidence (https://camarades.shinyapps.io/irise-soles/). To maintain a “living” evidence base, we automated the process, allowing for weekly updates. To ensure comprehensive coverage, we recently integrated grey literature sources, including preprints and conference abstracts. Our recently implemented feedback loop allows users to suggest corrections to automated screening decisions and annotations. These corrections can help us make continuous, community-driven improvements to the accuracy of our AI-driven screening and annotation process.
IRISE-SOLES provides the scientific community with a comprehensive, multi-disciplinary, up-to-date summary of interventions to improve reproducibility. The dashboard will allow researchers, policymakers and other stakeholders to make informed, evidence-based decisions on activities they undertake to improve reproducibility.
Accepted Paper
Short abstract
Long abstract
Reproducibility is fundamental to scientific progress. Multiple interventions to improve reproducibility have been proposed and/or tested, yet it remains unclear which strategies are most effective. As part of the iRISE (improving Reproducibility in SciencE) project, we have created a Systematic Online Living Evidence Summary (SOLES) to identify, curate, and visualise the entire literature base, aided by artificial intelligence (AI).
We systematically identified published articles describing interventions to improve reproducibility (n=16832 included). After dual-screening a subset (n=5000) for relevance, we trained a machine learning classifier to identify all relevant articles. We annotated 138 articles for predefined attributes, including scientific discipline, intervention, outcome, participants, and location. Using these annotations, we designed prompts and evaluated the annotation capabilities of different large-language models. The best performing approach was then applied across the included studies.
All outputs are presented on an interactive web dashboard, where users can interrogate the latest evidence (https://camarades.shinyapps.io/irise-soles/). To maintain a “living” evidence base, we automated the process, allowing for weekly updates. To ensure comprehensive coverage, we recently integrated grey literature sources, including preprints and conference abstracts. Our recently implemented feedback loop allows users to suggest corrections to automated screening decisions and annotations. These corrections can help us make continuous, community-driven improvements to the accuracy of our AI-driven screening and annotation process.
IRISE-SOLES provides the scientific community with a comprehensive, multi-disciplinary, up-to-date summary of interventions to improve reproducibility. The dashboard will allow researchers, policymakers and other stakeholders to make informed, evidence-based decisions on activities they undertake to improve reproducibility.
Synthezisers: metascience for meta-analysis
Session 1 Tuesday 1 July, 2025, -