Short abstract

The present study provides results of a research carried out in the Sao Paulo Research Foundation employing shadow experiments over thousands of reviews to understand the extent to which the selection is meeting the expected standards of excellence.

Long abstract

There is growing interest among research funding agencies worldwide in examining their selection processes to better understand whether these procedures meet the expected standards of excellence. This includes evaluating the outputs and outcomes of peer review, as well as investigating the potential presence of bias. Some funding agencies have pioneered small-scale experimental initiatives aimed at assessing the impact of alternative selection methods, in addition to traditional peer review, and determining whether such processes may introduce or mitigate selection bias. The present study provides both a review on these attempts and on the methodology and the results of a study carried out in the Sao Paulo Research Foundation (FAPESP). We conducted a shadow experiment to examine the differences between FAPESP’s historical peer review selection processes and alternative prioritization methodologies. The experiment was based on thousands of peer reviews, focusing on the possibilities of alternativa selection methods on the gray zone between the highest-ranked rejected proposals and the lowest-ranked accepted ones. In addition, we applied statistical methods to analyze the presence of potential biases in FAPESP’s selection procedures and to understand the parameters that influence policy making towards funding.