Peer review is now the predominant basis for decision-making in almost all forms of scientific output. This has been accompanied by scientific research into peer review. This panel builds on this and concentrates primarily on peer review in research grants.
Long Abstract
Peer review is seen as the gold standard of scientific decision-making, but it is also repeatedly criticized. Criticisms range from systematic bias effects, lack of replicability, status prevention, inhibiting innovations to overburdening the scientific system.
From a quantitative perspective, peer review is a successful model for decision-making in science. Until a few decades ago, peer review, applied primarily to publications and grants, was a quality assurance service provided by the Western scientific elite for the Western scientific elite. Today, peer review is universally demanded. Not only publications and grants are subject to peer review, but all kinds of scientific achievements and entities such as job appointments, research institutions, disciplines, funding programs, funding agencies, or entire countries.
As a result, scientific analysis with peer review has also rapidly gained importance. This panel aims to build on this, but concentrate primarily on research grants. Unlike publications, peer review of grants is subject to particular uncertainty, as a grant is a promise for the future. In recent years, a number of proposals have been made on how to deal with this uncertainty while keeping the workload for the scientific system within reasonable limits. These range from “fund people, not projects” and “collective allocation” to the use of random selection procedures.
Although this panel focuses primarily on peer review of grants, it covers a wide range of topics, from decisions made in various types of panels, the influence of gender and disciplines, and the use of AI for peer review.
We study negative potency, i.e. the tendency to give disproportionate weight to negative over positive information, in peer-review. While negativity is present at the individual level, it is mitigated during group deliberations, highlighting how groups mitigate individual biases.
Long abstract
We study negative potency, i.e. the tendency to give disproportionate weight to negative over positive information, in peer-review. While negativity is present at the individual level, it is mitigated during group deliberations, highlighting how groups mitigate individual biases.
We undertook a comparative investigation of the review behaviors and outcomes of nearly 300 experts who participated in the assessment of over 20 major S&T projects in 2024. This analysis was designed to elucidate the unique contributions of each type of expert and to identify distinct review patter
Long abstract
In practice, the participation of experts from diverse backgrounds in research assessment is increasingly prevalent. This trend is particularly evident in demand-oriented and large-invested science and technology (S&T) project reviews. In such contexts, experts from government management departments, market users, and other relevant sectors are often invited to join academic experts in forming a comprehensive review panel. The review process typically involves both quantitative scoring and qualitative evaluation, with experts providing detailed opinions to support their assessments.
Given this backdrop, it is essential to understand how experts from different backgrounds contribute uniquely to the research assessment process. Specifically, we aim to explore the distinct roles played by government management experts, user experts, and academic experts in project reviews. We are particularly interested in identifying any consistencies and differences among these groups in terms of their quantitative and qualitative evaluation opinions.To address these questions, we conducted a detailed statistical analysis and comparative study of the review behaviors and outcomes of nearly 300 experts who participated in the review of over 20 major S&T projects in 2024. Our analysis seeks to uncover the specific contributions of each type of expert and to identify patterns in their review behaviors and results.The findings of this study will provide valuable insights into the distinct roles and behaviors of experts from different backgrounds in research assessment. Moreover, our results will offer evidence-based recommendations for optimizing the use of expert input in future research assessments, thereby enhancing the effectiveness and fairness of the evaluation process.
Understanding the cost of developing, writing, reviewing and deciding on research proposals is a prerequisite to understanding the efficiency of funding decisions. This study estimated the cost and transaction cost for a sample of funding schemes and explored the qualitative benefits of the process.
Long abstract
Understanding the cost of developing, writing, reviewing and deciding on research proposals is a critical prerequisite to understanding the efficiency of funding decisions. The objective of this study was to develop estimates for cost and transaction cost – the ratio of cost to funding – for a sample of different funding schemes administered by two funders. The perceived benefits of the peer review funding process to applicants, reviewers and panellists were also explored. 12,617 people were surveyed between April 2022 and June 2023, with an overall response rate of 11%. The study estimated that the cost of grant and fellowship application processes is 13% of the value of the grant, and that 89% of those costs are borne by the applicants. This would suggest that any policy to increase the efficiency of grant funding (beyond increasing success rates) should focus on university and research institute practices, rather than those of research funders.
Using machine learning to analyze 39,458 peer review reports submitted to the national Swiss funder, we examined how gender and discipline relate to grant peer review content. We found substantive differences between female and male reviewers and across research fields.
Long abstract
Peer review by experts in the field is the cornerstone of the scientific review process. This study examined how gender and disciplinary norms shape the content and tone of grant peer review. We analyzed 39,458 review reports submitted to the Swiss National Science Foundation between 2016 and 2023, covering 11,409 applications for project funding across 21 disciplines from the Social Sciences and Humanities (SSH), Life Sciences (LS), and Mathematics, Informatics, Natural Sciences, and Technology (MINT). Using supervised machine learning, we classified over 1.3 million sentences by evaluation criteria and sentiment. Reviews in SSH were significantly longer and more critical, while those in MINT were more concise and positive. Female reviewers write longer and more structured reviews and are slightly more positive than their male counterparts. There were substantial differences across fields and disciplines, suggesting that disciplinary conventions strongly influence review practices. Our findings have implications for fairness, transparency, and consistency in research evaluation systems.
There are indications that grant review processes are conservative, disadvantaging novel research proposals. This study asks: What are the underlying mechanisms that might explain grant peer review's apparent conservatism?
Long abstract
The study is based on observations of 75 panels and interviews with 82 panellists, carried out over a 3-year period. The data set includes eight different funding competitions in five different research funding organisations in national, Nordic and EU settings. The quantity and variety that the empirical material covers are unparalleled by any previous qualitative studies of grant peer review.
There are indications that a complexity of mechanisms interact to produce conservative grant reviews. Panellists' tendency to focus on proposals' faults over their potential and their preference for sure over unsure outcomes leads to conservatism in individual assessments. Panel interaction in turn tends to strengthen individual panellists’ negativity dominance and risk avoidance, indicating group polarisation mechanisms are at play. However, panellists' negativity dominance is responsive to environment-level factors. The operationalisation of the scoring scale in instructions to reviewers aggravates negativity dominance when guidance equates top-scoring proposals with fault-free ones. Low success rates furthermore aggravates negativity dominance. In cases with low success rates, panellists tend to see their tasks as a rejection task, focusing on eliminating applications and thus on negative over positive aspects of proposals. A fault-focused scoring scale exerts a similar effect, reinforcing panellists' natural penchant to accord more weight to negative over positive aspects of proposals.
In conclusion, individual assessments of grant proposals favour safe and fault-free proposals. Environment-level factors and panel interaction reinforce rather than counteract these tendencies. Outcomes accordingly tend towards conservatism.
This large-scale replication study investigates AI’s use in evaluation, comparing AI and human output while assessing AI’s impact on reviewer output. We provide insights into reproducibility trends, AI-driven evaluation systems, and the future of on-demand replication."
Long abstract
As the quality of AI systems continues to improve and the costs of developing and deploying these models plummet, there has been a growing discourse about the role of AI in scientific discovery. This work investigates the use of AI in multidisciplinary scientific evaluation, with a particular focus on peer review and experimental replication. We conducted a large-scale replication study across various scientific disciplines, examining reproducibility trends and differences across domains. Alongside this, we implemented a large-scale AI-assisted peer review process, comparing the outputs of AI systems to those of human reviewers. This comparison not only highlights the strengths and limitations of AI in evaluating scientific work but also provides, to our knowledge, the first empirical evaluation of the impact of AI on human reviewers' quality. How does the presence of AI influence human judgment, confidence, and decision-making in the peer review process? Our findings offer novel insights into these questions, shedding light on the evolving dynamics between human and machine in scientific evaluation.
Furthermore, this study explores the potential for AI to serve as the foundation for a scientific evaluation "operating system", capable of streamlining and enhancing the peer review process. We discuss the implications of our results for the future of peer review, including the possibility of increased automation and the ethical considerations that accompany it. By bridging the gap between AI and human expertise, this work contributes to the broader conversation about the role of technology in shaping an optimistic future for scientific discovery and evaluation.
Short Abstract
Peer review is now the predominant basis for decision-making in almost all forms of scientific output. This has been accompanied by scientific research into peer review. This panel builds on this and concentrates primarily on peer review in research grants.
Long Abstract
Peer review is seen as the gold standard of scientific decision-making, but it is also repeatedly criticized. Criticisms range from systematic bias effects, lack of replicability, status prevention, inhibiting innovations to overburdening the scientific system.
From a quantitative perspective, peer review is a successful model for decision-making in science. Until a few decades ago, peer review, applied primarily to publications and grants, was a quality assurance service provided by the Western scientific elite for the Western scientific elite. Today, peer review is universally demanded. Not only publications and grants are subject to peer review, but all kinds of scientific achievements and entities such as job appointments, research institutions, disciplines, funding programs, funding agencies, or entire countries.
As a result, scientific analysis with peer review has also rapidly gained importance. This panel aims to build on this, but concentrate primarily on research grants. Unlike publications, peer review of grants is subject to particular uncertainty, as a grant is a promise for the future. In recent years, a number of proposals have been made on how to deal with this uncertainty while keeping the workload for the scientific system within reasonable limits. These range from “fund people, not projects” and “collective allocation” to the use of random selection procedures.
Although this panel focuses primarily on peer review of grants, it covers a wide range of topics, from decisions made in various types of panels, the influence of gender and disciplines, and the use of AI for peer review.
Accepted papers
Session 1 Tuesday 1 July, 2025, -