Concealing the identity of researchers and their institutions in the first assessment stage to reduce bias towards applicants
Paula Adam
(AQuAS)
Esther Vizcaino
(Agency for Health Quality and Assessment of Catalonia)
Maite Solans-Domènech
(Agency for Health Quality and Assessment of Catalonia (AQuAS))
Laura Puigcerver Navarro
(Agency for Health Quality and Assessment of Catalonia (AQuAS))
Paper Short Abstract
What is the effect of concealing researchers' and institutions' identities in a first evaluation stage in order to focus on the proposal's content? What external factors influence the decision-making process, regardless of the proposal's quality?
Paper Abstract
Multiple studies have shown that the main sources of bias in the peer-review process are related to the applicant's personal characteristics and institutional affiliation. We present our experience managing a project peer-review assessment process that includes three evaluation stages: first, individual peer-review assessment with blinded CVs; second, individual revision of the assessment with unblinded CVs by the same reviewer; and third, assessment of the highest-rated proposals by an ad hoc committee. Our research questions are: What is the effect of concealing researchers' and institutions' identities in the first evaluation stage? What external factors influence peer-reviewers' decision-making process? We address these questions through a retrospective observational study analyzing 5,002 individual proposal evaluations. The primary variable of analysis was the change between the first (blinded) and second (unblinded) assessments. We analyzed factors associated with changes, taking into account the characteristics of proposals, reviewers, and researchers. A qualitative content analysis was conducted to assess the reasons for changes. The analysis revealed that in 19% of the evaluations, reviewers changed their second assessment, either upgrading (12%) or downgrading (7%) their initial rating. Our findings suggest that changes in the second assessment were highly correlated with positive evaluations of the principal investigator's or research team's experience. These changes also seemed to be influenced by the total grant amount requested, the reviewer's world region, and the principal investigator's gender. Peer-review procedures that initially focus solely on the content of a research proposal offer a promising way to reduce some common biases related to researchers' characteristics.
Accepted Poster
Paper Short Abstract
Paper Abstract
Multiple studies have shown that the main sources of bias in the peer-review process are related to the applicant's personal characteristics and institutional affiliation. We present our experience managing a project peer-review assessment process that includes three evaluation stages: first, individual peer-review assessment with blinded CVs; second, individual revision of the assessment with unblinded CVs by the same reviewer; and third, assessment of the highest-rated proposals by an ad hoc committee. Our research questions are: What is the effect of concealing researchers' and institutions' identities in the first evaluation stage? What external factors influence peer-reviewers' decision-making process? We address these questions through a retrospective observational study analyzing 5,002 individual proposal evaluations. The primary variable of analysis was the change between the first (blinded) and second (unblinded) assessments. We analyzed factors associated with changes, taking into account the characteristics of proposals, reviewers, and researchers. A qualitative content analysis was conducted to assess the reasons for changes. The analysis revealed that in 19% of the evaluations, reviewers changed their second assessment, either upgrading (12%) or downgrading (7%) their initial rating. Our findings suggest that changes in the second assessment were highly correlated with positive evaluations of the principal investigator's or research team's experience. These changes also seemed to be influenced by the total grant amount requested, the reviewer's world region, and the principal investigator's gender. Peer-review procedures that initially focus solely on the content of a research proposal offer a promising way to reduce some common biases related to researchers' characteristics.
Poster session
Session 1 Tuesday 1 July, 2025, -