Paper short abstract:

The study explores an intervention aiming to enact ‘responsible use of metrics’ in peer review of potential nominations for high-stakes scientific prizes. It focuses on how the removal of some quantitative indicators influences valuation practices of the assessment committee.

Paper long abstract:

Initiatives to reform research assessment have for at least a decade strived to reposition how academics are evaluated. The resulting interventions in evaluative practices influence what counts as quality in academia.

A particularly pertinent facet of the reforms concerns promoting responsible use of metrics or decoupling research assessments from quantitative indicators altogether. One of the objectives is to shift understandings of quality from one that is quantitatively proxied (e.g. through number of publications or citations) to one rooted in broader values or more inherent in the content of the work. The resulting notion of quality is therefore often aimed to be de-quantified.

In this paper, I explore the results of an intervention aiming to enact ‘responsible use of metrics’. The case study concerns an internal assessment at a Dutch university in which candidates for high-stakes national prizes are evaluated to decide on the university's nominees. Some quantitative indicators were removed from that process due to concerns around responsible evaluation.

The case offers insight into how the removal of systematically provided indicators influences valuation practices of the assessment committee. How do peer reviewers adapt to this intervention and through which strategies do they construct quality and reach their evaluative decisions?

Theoretically the case explores implications of a shift away from metrics for calculation (Callon and Muniesa, 2005) and qualculation (Cochoy, 2002; Callon and Law, 2005) as processes of evaluative decision making. It broadens our understanding of how calc/qualculative rationality works after intervention in calculative agencies in a research assessment reform context.