Paper short abstract:

Our contribution relates researchers' quality conceptions with their (e-)valuation of citation- and web-based metrics. We then contrast these views with an expert discourse on Altmetrics that hopes to overcome problems in research evaluation by expanding the range of research metrics.

Paper long abstract:

Recent years have witnessed a revival of the old question what research quality is and how best to assess it. A particular concern revolves around numerical indicators and what they (should) measure. A repeated suggestion is to tie indicators to what the assessed perceive as quality. These debates have been propelled by the occurrence of new metrics and their intermingling with novel value concepts. One example is the propagation of Altmetrics and their conjunction with notions of openness and societal impact.

Our contribution firstly relates researchers’ understandings of scientific quality to their (e-)valuation of metric indicators. Secondly, these appraisals are juxtaposed to experts’ visions of evaluation reform through web-infrastructures and novel online data. We conducted 25 semi-structured interviews with German-affiliated researchers from Genetics and Psychology. Respondents reflected on how they assessed research value, on the quality of one of their journal articles, and on the meaning of metrics attached to this paper (citations, Altmetrics, JIF). These perspectives are contrasted with 14 expert interviews about Altmetrics’ perils and potentials.

Both researchers and experts criticize reactive effects of existing indicators. Researchers value indicators that reflect scientific chains of production. Accordingly, they judge few Altmetrics as valuable, most as irrelevant, and some as potentially emulating established indicators’ shortcomings. Despite Altmetrics limitations, experts value concomitant datafication-infrastructuration processes as potentially overcoming problems in research assessment: Exploring new kinds of impact, aligning evaluation with newly crafted goals, incentivizing altered behaviors: Altmetrics exemplify a current deterritorialization of research and its reterritorialization along the lines of platform capitalism.