Paper long abstract:

Non- and misuses of professional bibliometric indicators has become a matter of concern for scientometrics in recent times, with the growing perceptions of widespread uses of alternative, ready-to-hand products in formal research evaluation contexts and academic life more generally (Buela-Casel & Zich, 2012, Derrick & Gillespie, 2013). As well as a growing source of competition, some in the community see the widespread presence of 'amateur' bibliometrics as a threat to the credibility of quantitative performance measures per se (Wouters et al, 2014). To date one key mediating strategy of professional scientometricians has been articulating standards for using advanced bibliometrics in research evaluation, through producing and circulating initiatives like best practice guidelines. This paper is about problems in 'implementing' such standards (Zuiderent-Jerak, 2007). Specifically we follow ongoing struggles to transport standards from 'knowledge infrastructures' into 'evaluation infrastructures' (Wouters, 2014) in Dutch University Medical Centers (UMCs).

First we draw on fieldwork carried-out among six research groups in UMCs and two evaluation panels charged with formally evaluating the performance of a single UMC, along with interviews with UMC administrators. We report on key moments of 'data friction' and 'infrastructural inversion' (Edwards, 2010) vis-à-vis situating professional standards and measures. Through document analysis we then argue 'distancing work' inscribed in scientometic 'meta-data' (ibid) are generally inhibiting successful extensions of professional knowledge infrastructures into evaluation practices. We conclude that shifts towards 'located' (Suchman, 2002) and 'situated' (Engel & Zeiss, 2013) accountability practices would constitute a more promising strategy for 'implementing' standards within the research evaluation contexts.