Paper short abstract:

A survey of 485 life scientists engaged in research assessment finds gaps between the importance placed on assessing the credibility of research outputs and satisfaction with their ability to do so. These results suggest opportunities to develop better indicators of credibility and research quality.

Paper long abstract:

Researchers serving on grant review and hiring committees make high-stakes decisions about the quality of candidates’ research. Under conditions of limited time and attention, they may resort to journal-based impact metrics or reputational judgments as extrinsic proxies for research quality. These evaluative practices have been criticized by reform initiatives like the Coalition for Advancing Research Assessment (CoARA) because, inter alia, they do not address intrinsic characteristics of the evaluated research. Building on a previous interview study and on STS work on indicators, we surveyed 485 biology researchers who recently served on a research assessment committee to better understand their practices and priorities in the committee context. We found that assessing credibility or trustworthiness is very important to most (81%) respondents’ evaluations in this setting, although fewer than half of respondents were satisfied with their ability to assess credibility. We found similar gaps for specific evaluative tasks, particularly around the assessment of research integrity and of transparency in reporting. While a substantial proportion (57%) of respondents acknowledged using Journal Impact Factor and journal reputation to assess credibility, our results suggest opportunities to develop better indicators or signals to support the evaluation of credibility and, by extension, research quality. Even as such signals are proxies in their own right, we argue that they can usefully supplement personal inspection in a less distortionary way than journal-based measures and thereby align with notions of research quality as an attribute of the work itself, rather than its container within the published scholarly record.