Automated interpretation of statistical tables in economics: prevalence of reporting errors and effectiveness of open science policies
Stephan Bruns
(Hasselt University)
John Ioannidis
(Stanford University)
Fabian Henrik Carl Raters
(University of Göttingen)
Chris-Gabriel Islam
(COGITANDA)
Short abstract
We developed a tool that automatically extracts statistical values from tables (DORIS), resulting in a sample of 578,132 statistical tests from economics. We analyze (1) the prevalence of statistical reporting errors and (2) the effects of data and code availability policies.
Long abstract
We developed a tool that automatically extracts statistical values from tables (DORIS), resulting in a sample of 578,132 statistical tests from the top 50 economics journals (1998-2016). We analyze (1) the prevalence of statistical reporting errors that occur if the eye-catcher depicting the level of statistical significance is inconsistent with the reported statistical values. 14.88% of the articles have at least one strong reporting error in the main tests and there is a bias toward statistical significance. We also analyze (2) the effects of data and code availability policies on characteristics of published articles. We use a staggered difference-in-differences design, showing that these policies result in slightly less emphasis on statistical significance, more rigor in reporting, fewer reporting errors in control variables, but not in more citations.
DORIS can be used in the review process to improve article quality and to generate data at a large scale for future meta-research.
Accepted Paper
Short abstract
Long abstract
We developed a tool that automatically extracts statistical values from tables (DORIS), resulting in a sample of 578,132 statistical tests from the top 50 economics journals (1998-2016). We analyze (1) the prevalence of statistical reporting errors that occur if the eye-catcher depicting the level of statistical significance is inconsistent with the reported statistical values. 14.88% of the articles have at least one strong reporting error in the main tests and there is a bias toward statistical significance. We also analyze (2) the effects of data and code availability policies on characteristics of published articles. We use a staggered difference-in-differences design, showing that these policies result in slightly less emphasis on statistical significance, more rigor in reporting, fewer reporting errors in control variables, but not in more citations.
DORIS can be used in the review process to improve article quality and to generate data at a large scale for future meta-research.
Methods mash: expanding the tools of metascience
Session 1 Tuesday 1 July, 2025, -