Paper short abstract:

As collective frames of scientific communities, research quality notions are difficult to identify because actors are only partly aware of them and enact them in the language of their field. I present an approach for the empirical identification of quality notions that addresses these challenges.

Paper long abstract:

If we define notions of research quality as collective frames of scientific communities for the assessment of the utility of scientific contributions, methodological challenges arise because as cognitive structures actors are only partly aware of, frames are notoriously difficult to detect. Furthermore, the enactment of such frames in interactions between researchers must be analysed, which requires interpreting quality judgments uttered by researchers in the language and context of their research. The common approach of co-constructing quality notions in interactions between researchers and their STS observers, e.g. by letting researcher specify general quality criteria like originality cannot ascertain how quality frames are enacted in situ.

I will present an another qualitative approach, which I applied for identifying quality notions used by researchers in three specialties (plant biology, medieval history, ultrafast physics) when assessing the work of their colleagues. Quality judgments were investigated in formal assessment situations (peer review of publications and grant proposals) as well as informal assessment situations (seminar and conference discussions), using observations, content analysis of review documents, and interviews with researchers. This general strategy was tailored to the communication practices of each research field. From the arguments made by researchers, I distilled field-specific quality criteria with bottom-up coding and compared them across various assessment situations to arrive at collectively shared notions of research quality. I will discuss the methodological challenges of this approach.