Short abstract

A trial of Distributed Peer Review (DPR) by the Volkswagen Foundation is reported. Quantitative and qualitative analysis is applied to positives and negatives of the DPR process, and the contrast to the results of traditional panel review.

Long abstract

Funders report increasing difficulties in the timely identification and allocation of application reviewers. Distributed Peer Review (DPR), where applicants also serve as reviewers on the same call, is a potential solution. VolkswagenStiftung (VWS) conducted an experiment trialling DPR alongside traditional panel review for the 2024 round of their humanities and social sciences call ‘Open Up’, with a commitment to funding proposals selected via both routes.

A mixed methods evaluation of this experiment was conducted by independent researchers from The Research on Research Institute (RoRI). Overlap between DPR and panel review processes was examined and the stability and efficiency of the DPR process assessed. Interviews were carried out to explore the experiences of funders, panel reviewers and applicants.

140 proposals were submitted and assessed; 11 were selected by panel and 10 by DPR to receive funding. Three proposals were selected via both routes.Evaluation of DPR allowed for unusual levels of insight into reviewer consistency and reasoning. Applicants identified both positives and negatives of the DPR process.

The results support the potential for DPR as an acceptable, efficient and robust method of funding evaluation. However, it differs in some important ways from more ‘traditional’ panel-based selection methods, meaning that outcomes of DPR processes may also differ. We discuss tensions between the “senior gatekeeper” (panel) and “wisdom of the expert crowd” (DPR) models, as well as additional benefits of DPR in terms of legitimacy, aligning reviewers with funding call criteria and widening and diversifying the pool of people who review funding proposals.