Short abstract

Medical systematic reviews follow a rigorous process and assess the quality of evidence using the GRADE framework. We gathered 60,000 gradings of evidence of 6,300 Cochrane systematic reviews published over 15 years. Overall, the level of evidence was high in only 7% with no improvement over time.

Long abstract

Context: Systematic reviews and meta-analyses are essential to support decision making. In the context of health research, the Cochrane Database of Systematic Reviews (CDSR) is renowned for its rigorous synthesis of evidence and its updating process as new evidence emerges. Cochrane implements the GRADE framework and reports the certainty of evidence—high, moderate, low, very low— for all prespecified primary outcomes in a summary of finding tables. Our study aims to advance metascience by elucidating trends in evidence quality using large-scale systematic reviews data.

Methods: We identified all Cochrane systematic reviews indexed in CDSR from 2010 to 2025 that reported a summary finding table, and extracted certainty ratings using chatgpt-4o-mini. A quality assurance process of data extraction is ongoing. All data and codes will be shared on Github and on an associated Zenodo repository with a DOI.

Results: We identified 6,346 systematic reviews, yielding data on 59,979 graded medical outcomes. Our preliminary analyses reveal the following distribution of evidence certainty: 7% high, 24% moderate, 37% low, and 33% very low. Although preliminary results suggest a trend toward worsening certainty over time, these findings may be subject to residual confounding. To mitigate such effects, we will specifically assess the evolution of certainty in updates of reviews addressing the same research questions.

Conclusion: Our results should question how primary research is planned and conducted and the role of systematic reviews for improving primary research.