Our aim is to catalyze new experimental collaborations between researchers, funders, publishers and institutions interested in generating robust evidence on how to improve research. Showcasing existing experiments and matchmaking around new proposal development. Let’s get experimental!
Long Abstract
Organised in partnership with Open Philanthropy and RoRI’s AFiRE programme, these three linked sessions will facilitate matchmaking and networking for experimentation across all areas of metascience, with a focus on interventions to support higher quality, lower cost and more impactful research.
Each session will showcase metascience principles, methods or examples of experimentation, as well as providing a platform for co-developing new project ideas by participants. Researchers, funders, universities, publishers and other actors in the research ecosystem are invited to propose experiments and matchmake with potential collaborators.
The Abundance and Growth Fund at Open Philanthropy is happy to consider proposals that emerge from this process, with an aim to fund pilot work on the most promising opportunities for generating credible evidence to support improvements to the research system. Projects with a strong analysis strategy, whether experimental, quasi-experimental or exploratory, will be favoured. Facilitation will be provided by RoRI’s AFIRE programme. Let’s get experimental!
A trial of Distributed Peer Review (DPR) by the Volkswagen Foundation is reported. Quantitative and qualitative analysis is applied to positives and negatives of the DPR process, and the contrast to the results of traditional panel review.
Long abstract
Funders report increasing difficulties in the timely identification and allocation of application reviewers. Distributed Peer Review (DPR), where applicants also serve as reviewers on the same call, is a potential solution. VolkswagenStiftung (VWS) conducted an experiment trialling DPR alongside traditional panel review for the 2024 round of their humanities and social sciences call ‘Open Up’, with a commitment to funding proposals selected via both routes.
A mixed methods evaluation of this experiment was conducted by independent researchers from The Research on Research Institute (RoRI). Overlap between DPR and panel review processes was examined and the stability and efficiency of the DPR process assessed. Interviews were carried out to explore the experiences of funders, panel reviewers and applicants.
140 proposals were submitted and assessed; 11 were selected by panel and 10 by DPR to receive funding. Three proposals were selected via both routes.Evaluation of DPR allowed for unusual levels of insight into reviewer consistency and reasoning. Applicants identified both positives and negatives of the DPR process.
The results support the potential for DPR as an acceptable, efficient and robust method of funding evaluation. However, it differs in some important ways from more ‘traditional’ panel-based selection methods, meaning that outcomes of DPR processes may also differ. We discuss tensions between the “senior gatekeeper” (panel) and “wisdom of the expert crowd” (DPR) models, as well as additional benefits of DPR in terms of legitimacy, aligning reviewers with funding call criteria and widening and diversifying the pool of people who review funding proposals.
This paper examines the impact of the British Academy's switch to a conditional lottery for allocating research funding in its Small Grant Scheme. Findings show a significant overall increase in applicants, with a disproportionately higher rise among Asian and British Asian applicants.
Long abstract
In 2022, the British Academy (BA) began trialling a conditional lottery to allocate research funding through its Small Grant Scheme. The aim was to determine whether using a lottery-based allocation system could reduce biases and, in turn, increase the diversity of grant recipients.
This paper examines the impact of replacing a traditional peer review system for awarding small grants with a system in which grants are allocated randomly, provided they meet a minimum quality threshold. Specifically, it analyses how partial randomisation influences applicants' characteristics.
Using data from the BA on applicants to the Small Grant Scheme, Postdoctoral Fellowship, and Mid-Career Fellowship—both before and after the introduction of partial randomisation—this paper employs a difference-in-differences design to assess whether applicant characteristics changed following the shift to a lottery-based allocation. We examine the number and proportion of applicants across various demographic and institutional characteristics.
Our findings indicate a substantial overall increase in the number of applicants, with approximately 500 additional individuals applying to the scheme each year. The number of applicants increased across all demographic categories analysed and rose in both Russell Group and non-Russell Group universities. However, there was a disproportionate rise in the number of Asian and British Asian applicants, while the number of white applicants declined disproportionately. These results have important implications for the role of partial randomisation in promoting ethnic diversity in research funding. They also highlight the potential for broadening the "diversity of ideas" in research by expanding the applicant pool.
We present a method and preliminary results for measuring scientific impact of funded vs. unfunded applicants from the Gordon and Betty Moore Foundation Experimental Physics Investigator initiative.
Long abstract
Measuring scientific impact is at the heart of the question, ‘Do people who receive grant funding have more scientific impact than their equal-potential counterparts who do not get funded?’ In partnership with the Gordon and Betty Moore Foundation Experimental Physics Investigator (EPI) Initiative, we present a method and preliminary results for measuring scientific impact of funded vs. unfunded applicants. The method draws on the network normalized citation index, Ĉ, (Ke, Gates, Barabasi, 2023). This citation index addresses concerns common to other citation indices, particularly that different fields have different publication practices and that citation counts simply increase as a natural course of time. The citation index, Ĉ, compares papers to others in the same subdiscipline not by determining a priori the subdiscipline, but instead by creating a citation network and then using the average yearly citations of nearest neighbors to normalize. Because Ĉ uses average yearly citations, it also accounts for the tendency of citations to increase over time. We present a preliminary analysis of the 2022 cohort of the EPI Initiative, as we intend to follow a similar analysis pattern for other cohorts in a pre-registered study of scientific impact. The 2022 cohort included 16 funded EPI investigators and 8 members of a comparison group (i.e., equal-potential, unfunded applicants). This analysis is based on the set of papers from the 2022 cohort of EPI investigators (N = 734) and comparison group members (N = 434).
Tom Stafford
Short Abstract
Our aim is to catalyze new experimental collaborations between researchers, funders, publishers and institutions interested in generating robust evidence on how to improve research. Showcasing existing experiments and matchmaking around new proposal development. Let’s get experimental!
Long Abstract
Organised in partnership with Open Philanthropy and RoRI’s AFiRE programme, these three linked sessions will facilitate matchmaking and networking for experimentation across all areas of metascience, with a focus on interventions to support higher quality, lower cost and more impactful research.
Each session will showcase metascience principles, methods or examples of experimentation, as well as providing a platform for co-developing new project ideas by participants. Researchers, funders, universities, publishers and other actors in the research ecosystem are invited to propose experiments and matchmake with potential collaborators.
The Abundance and Growth Fund at Open Philanthropy is happy to consider proposals that emerge from this process, with an aim to fund pilot work on the most promising opportunities for generating credible evidence to support improvements to the research system. Projects with a strong analysis strategy, whether experimental, quasi-experimental or exploratory, will be favoured. Facilitation will be provided by RoRI’s AFIRE programme. Let’s get experimental!
Accepted papers
Session 1 Tuesday 1 July, 2025, -