Analysing the Reliability of Quantitative Impact Evaluations (ARGIE)
Jack Blumenau
(University College London)
Anouk Rigterink
(Durham University)
Paper Short Abstract
We conduct a series of “many-teams” analyses on a variety of government impact evaluations to understand analysis-dependence in government research. By partnering with social researchers from the UK Cabinet Office, the findings from our project will directly inform research practice in government.
Paper Abstract
How reliable are the impact evaluations conducted by government? Impact evaluations – studies which measure the effects of policy interventions – play a central role in government decision-making, and UK government departments spend hundreds of millions of pounds on conducting and commissioning evaluation activities each year.
Recent metascience research demonstrates that many quantitative research findings are highly analysis-dependent: researchers investigating the same research question, using the same data, often make radically different analytical choices, and draw highly contrasting conclusions. If the estimated effects of a policy intervention vary substantially with analysts’ choices, we cannot be sure that the information from any single analysis that feeds into policymaking is correct. Understanding how sensitive government impact evaluations are to analysis decisions is therefore crucial for improving the robustness of evidence-based policymaking.
Our project, which is funded through a UKRI Metascience Research Grant, aims to generate new evidence on the robustness of quantitative research in government and provide actionable guidance on ameliorating analysis dependence for government researchers. First, we build on recent metascientific work and conduct a series of “many-teams” analyses to re-evaluate the estimates of a broad variety of existing government impact evaluations. Second, our project unites an academic team with government social researchers in the UK Cabinet Office, which enables us to translate our findings into new guidance on mitigating analysis-dependence which will be incorporated into the Magenta Book, a key resource for all evaluators across UK government (due to be published in 2025).
Accepted Poster
Paper Short Abstract
Paper Abstract
How reliable are the impact evaluations conducted by government? Impact evaluations – studies which measure the effects of policy interventions – play a central role in government decision-making, and UK government departments spend hundreds of millions of pounds on conducting and commissioning evaluation activities each year.
Recent metascience research demonstrates that many quantitative research findings are highly analysis-dependent: researchers investigating the same research question, using the same data, often make radically different analytical choices, and draw highly contrasting conclusions. If the estimated effects of a policy intervention vary substantially with analysts’ choices, we cannot be sure that the information from any single analysis that feeds into policymaking is correct. Understanding how sensitive government impact evaluations are to analysis decisions is therefore crucial for improving the robustness of evidence-based policymaking.
Our project, which is funded through a UKRI Metascience Research Grant, aims to generate new evidence on the robustness of quantitative research in government and provide actionable guidance on ameliorating analysis dependence for government researchers. First, we build on recent metascientific work and conduct a series of “many-teams” analyses to re-evaluate the estimates of a broad variety of existing government impact evaluations. Second, our project unites an academic team with government social researchers in the UK Cabinet Office, which enables us to translate our findings into new guidance on mitigating analysis-dependence which will be incorporated into the Magenta Book, a key resource for all evaluators across UK government (due to be published in 2025).
Poster session
Session 1 Tuesday 1 July, 2025, -