Our aim is to catalyze new experimental collaborations between researchers, funders, publishers and institutions interested in generating robust evidence on how to improve research. Showcasing existing experiments and matchmaking around new proposal development. Let’s get experimental!
Long Abstract
Organised in partnership with Open Philanthropy and RoRI’s AFiRE programme, these three linked sessions will facilitate matchmaking and networking for experimentation across all areas of metascience, with a focus on interventions to support higher quality, lower cost and more impactful research.
Each session will showcase metascience principles, methods or examples of experimentation, as well as providing a platform for co-developing new project ideas by participants. Researchers, funders, universities, publishers and other actors in the research ecosystem are invited to propose experiments and matchmake with potential collaborators.
The Abundance and Growth Fund at Open Philanthropy is happy to consider proposals that emerge from this process, with an aim to fund pilot work on the most promising opportunities for generating credible evidence to support improvements to the research system. Projects with a strong analysis strategy, whether experimental, quasi-experimental or exploratory, will be favoured. Facilitation will be provided by RoRI’s AFIRE programme. Let’s get experimental!
This study examines the leverage effect of KU Leuven’s internal grants on securing external funding. A counterfactual analysis matched funded and non-funded professors, revealing a significant boost for those with low initial budgets.
Long abstract
KU Leuven's internal funding aims to support researchers in successfully applying for subsequent regional, national or international grants.
This study investigates if indeed KU Leuven's internal project grants act as a leverage on the acquisition of external funding. A counterfactual analysis compared two groups of professors: those who received an internal project (target group) and those who did not (control group). Researchers were matched using Nearest Neighbour analysis based on various categorical variables, correcting for the Matthew effect, resulting in 198 matched pairs. The methodology applied overcomes the difficulty of assessing the net effect of one particular funding source in a heterogenous funding landscape.
Paired t-tests or Wilcoxon tests with Bonferroni correction and difference-in-difference analysis indicated a significant leverage effect of the internal grants for researchers with starting budgets from other sources below 120 k€, while no significant effect was observed for those with higher initial budgets. Spearman correlation analysis showed negligible or weak correlations between initial budget size and external funding, except for the highest budget class.
The findings suggest that the internal grants effectively enhance external funding acquisition for researchers with limited initial budgets, though caution is advised due to potential unaccounted factors.
We report the results of a shadow experiment on proposals to the Metascience 2025 conference. After programme selection was finalised, we evaluated the use of large language models to successfully predict which reviewers would judge themselves most suitable to review submitted proposals.
Long abstract
Peer review is central to the evaluation of research, not least at this conference where all submitted proposals are reviewed by a subset of the programme committee. Identifying and recruiting suitable reviewers is a key bottleneck in the peer review process. AI tools promise to be able to match texts to reviewers, and so improve the efficiency (directly) and quality (indirectly) of reviews. By conducting a “shadow experiment” on the Metascience 2025 submissions we seek to evaluate this potential. After selection for the conference has been completed, including reviewers scoring proposals and their own suitability to review, we will investigate the feasibility of running, in-house, privacy-respecting, language models which are capable of matching reviewer profiles to proposal texts. The ambition will be to calculate match scores for all possible reviewer-proposal combinations and so be able to analyse a) how well the actual matching was against the suggested optimum and b) how well AI matching is able to predict self-rated reviewer suitability to review. Reporting the results at the conference will contribute to Metascience as a self-reflective and learning organisation.
Tom Stafford
McKenzie Leier (Massachusetts Institute of Technology)
Short Abstract
Our aim is to catalyze new experimental collaborations between researchers, funders, publishers and institutions interested in generating robust evidence on how to improve research. Showcasing existing experiments and matchmaking around new proposal development. Let’s get experimental!
Long Abstract
Organised in partnership with Open Philanthropy and RoRI’s AFiRE programme, these three linked sessions will facilitate matchmaking and networking for experimentation across all areas of metascience, with a focus on interventions to support higher quality, lower cost and more impactful research.
Each session will showcase metascience principles, methods or examples of experimentation, as well as providing a platform for co-developing new project ideas by participants. Researchers, funders, universities, publishers and other actors in the research ecosystem are invited to propose experiments and matchmake with potential collaborators.
The Abundance and Growth Fund at Open Philanthropy is happy to consider proposals that emerge from this process, with an aim to fund pilot work on the most promising opportunities for generating credible evidence to support improvements to the research system. Projects with a strong analysis strategy, whether experimental, quasi-experimental or exploratory, will be favoured. Facilitation will be provided by RoRI’s AFIRE programme. Let’s get experimental!
Accepted papers
Session 1 Wednesday 2 July, 2025, -