Paper short abstract:

The objective of this paper is to examine how breakthrough research accomplishments are created under these accelerated conditions. Through interviews with researchers behind breakthrough results, we will seek to characterize research processes and how the breakthroughs came about.

Paper long abstract:

The conditions under which academic researchers operate have experienced rapid change. Dependence on competitive funding has increased considerably, combined with greater concentration into larger grants (Bloch and Sørensen, 2015). Alongside this, the distribution of institutional block funding is increasingly attached to research performance, often based on quantitative measures (Hicks, 2011). At the same time, the number of PhDs has increased dramatically, resulting in intense competition for both temporary and tenured positions (Fochler et al, 2015).

Arguments have been made that these factors can have important effects on researcher behavior, and on how excellent or breakthrough research is produced (Hammarfelt and de Rijcke, 2014; Hicks et al, 2015; Münch, 2014). However, it is not fully clear how these new conditions for research influence breakthrough research (Heinze et al, 2009; Laudel and Gläser, 2014). In what ways is the actual pursuit of scientific excellence adversely affected? Or is it more of a question of adaptation to the new reality of research? Is breakthrough research generally unaffected by these changed conditions?

The objective of this paper is to examine how breakthrough research accomplishments are created under these accelerated conditions. Through interviews with researchers behind breakthrough results, we will seek to characterize research processes and how the breakthroughs came about. The interviews will in particular focus on potential influences of changed conditions on risk taking, research ambitions and time horizons, and the role of serendipity and the pursuit of unexpected results.

Paper short abstract:

In competitive excellence, is failure becoming a taboo? How does this affect research? We compare cultural ways of dealing with uncertainties in the academic life sciences and biotech companies and ask which role the chance of failure plays in epistemic, organizational and career decisions.

Paper long abstract:

In science policy, funding and academic research practice, excellence is ever more strongly defined in terms and processes of competition. For the most part, the truly excellent are those with a track record repeatedly proving their ability to outrun others. Particularly in "hot" fields, failing to secure a grant, failing to achieve significant results in an experiment, failing to publish a paper with high impact, is seen as most likely leading to career death, at least in specific sensitive phases of academic lives. As a result, failure, or even the possibility of failing, is culturally side-lined and may even have become a taboo. The consequences of this for research as an activity which by its very definition needs to deal with uncertainty are only starting to be discussed, for example in debates about missing opportunities to publish negative results or about the manipulation of research results.

In this paper, we will discuss how uncertainty and failure are defined and handled in two very different institutional contexts in the contemporary life sciences: academia and biotechnology companies. Drawing on interview data, we will analyse and compare different cultural ways of dealing with uncertainties and ask which role the possibility of failure plays in making epistemic, organizational and career-related decisions.

We explore the differing cultural roles of failure to better understand how discourses about excellence re-shape researchers' tactical decisions in research and research careers in academia, but also to address alternative ways of dealing with the inherent uncertainties of science.

Paper short abstract:

This paper discusses how practices of evaluation in Dutch law schools perform ‘academic quality’ and mediate research practices. Based on interviews with researchers and administrators and observations of evaluation practices, this paper contributes to the study of the social lives of indicators.

Paper long abstract:

This paper discusses how practices of evaluating academic work in Dutch law schools contribute to the constitution of notions of 'academic quality', 'public relevance' and 'the excellent researcher'. Since the 1980s, quantitative performance measures have become ubiquitous, including in academia. Various formats of measuring the performance of individuals, groups and institutions, have evolved as disciplinary techniques of government, contributing to the constitution of 'kinds of academics' and mediating research practices. Paradoxically, researchers are, forcibly and ambiguously, implicated in the manifestation of the evaluation machinery themselves. While the use of quantitative indicators to assess academic worth is widely critiqued for relying on artificial proxies and generating perverse effects, simultaneously universities, faculties and researchers contribute to their legitimacy by maintaining an interest in high scores, by using results for management, policy and communication and by participating in the adaptation of evaluation practices and performance management systems. Taking into account that practices of evaluation are both an effect and constitutive of a normative logic of worth, which is contested, diverges between and within research fields and requires a continuous work of enacting 'agreements', hermeneutically as well as materially, the central question this paper addresses is how this state of agreement is reached in practice. Based on interviews with researchers and administrators in Dutch law schools, as well as observations of the production of specific formats and bibliometric indicators used in research assessments, this paper will contribute to a further investigation of the material and social lives of indicators in evaluation regimes.

Paper short abstract:

The contribution will unpack notions of scientific governance in analysing the rhetoric in academia about ‘performance’ based on case studies about performance measurement. This contributes to STS in unwrapping rhetoric of performance as something that merges old symbols and new master terms.

Paper long abstract:

The contribution will unpack notions of scientific governance by taking the rhetoric in academia about 'performance' as analytical object. University staff seems to be stressed by several processes of management, like evaluations and competitions. Whatever arises from those solutions, the new-academic language of excellence is one of those structural effects.

Public debate about scientific governance is swayed by propaganda of the new governance as well as critical defence against an invasion of further economization. Both seem to be arrived at a stalemate. In contrast to the best known pro-con-arguments our aim is to show what can be rescued by listening to these narrations of (de-)justifications of governing excellence.

Based on case studies from a research project about performance measurement in German universities we investigate the descriptions made in interviews on whether these systems work. We will present a) dominant dichotomies and their recurring distinctions,

b) most addressed points of reference and c) metaphors to discuss the measurability of (personal) scientific performance.

This will give insight into the mindsets of orientations and its spheres, which donate images to science (like economy, nature, family, traffic). The rhetoric can be analysed as maintenance of a fictional consensus about the conventions of representing peoples' scientific capabilities. The input would contribute to STS literature in unwrapping the rhetoric of performance as something that merges old symbols (like alma mater) and new master terms (like competition).

As alternative forms of interactions are welcomed we consider presenting the input as a guessing game or collective crossword puzzle.

Paper short abstract:

Focusing on the experiences of foreign scientists in Japan’s research institutions, the paper examines excellence discourses from the perspective of labor, highlighting the lived realities and the human cost of excellence-oriented science policies.

Paper long abstract:

Along with science policy makers around the world, the Japanese government has embraced the rhetoric of research excellence in recent decades. Internationalization of research is perceived as a crucial facet of the national project to improve the country's scientific institutions and ensure excellence. Importantly, recent science and technology policies in Japan also posit development of "excellent human resources" as the basis of scientific innovation. One of the means of achieving this goal is the recruitment of international researchers in temporary positions.

However, as participants in the increasingly transnational circulation of academic knowledge workers, the foreign scientists enlisted to advance Japan's research sector encounter considerable uncertainties in their work and personal lives. Based on ethnographic research with early career life scientists in Japan's Kansai region, my paper examines the dissonance that emerges between discourses of excellence and the lived realities of those who are recruited to produce it. Focusing on the ways young foreign researchers make sense of being scientists at Japanese institutions, I suggest that, first, researchers engage in highly personalized forms of emotional labor in order to retain employment and, second, they explain these investments as a feature of a peculiarly Japanese framework of excellence, thus disguising the increasing demands of the transnational scientific labor market. In line with the panel's goal of unpacking the notion of excellence, the contribution of my paper to the STS literature lies in its aim to examine and problematize the human cost of research excellence discourses.

Paper short abstract:

In Slovenia, quantitative indicators are increasingly used to evaluate scientific excellence. We use bibliometric analysis and interviews with excellent scientists to explore the effects such a trend is having on the research performance and practices in the small Slovenian scientific community.

Paper long abstract:

The evaluation of scientific performance in contemporary science systems is increasingly relying on quantitative indicators of excellence, determined by national science policies. We explore the conceptualization and the effects of quantitative measures of research excellence on the small scientific community in Slovenia. For this purpose, we combine the results of a bibliometric analysis with the results of interviews among excellent Slovenian scientists.

Our aim is to contribute to a better understanding of the effects of such excellence policies as well as of the factors that influence their results.

In our quantitative analysis, we focus on research excellence as defined in the research evaluation methodology of the Slovenian Research Agency, and explore its effects on the performance of Slovenian scientists in different scientific fields. We also investigate how science policy mechanisms, such as competitive funding and research collaboration affect the quantitative indicators of excellence. Here a multilevel analysis using a hierarchical linear model with regression analysis is applied to the data with several nested levels. In the qualitative part, we present the results of the interviews with excellent scientists. We focus on their views of what makes up scientific excellence, which policy factors are important for promoting it, and whether they consider quantitative indicators as a proper instrument for promoting excellent science.

Thus we hope to provide a more comprehensive view of the quantification of excellence in Slovenia and internationally, its effects on research performance, what role competitive funding and collaboration play in it, and how scientists themselves perceive its effects.

Paper short abstract:

This paper empirically tests for differences in the novelty of funded outputs between internal block and competitive project funded papers. Findings suggest competitive project selection procedures are less receptive to novel ideas from researchers with low academic status.

Paper long abstract:

In many countries the scientific funding system is shifting from an internal block funding model towards a competitive project funding one. However, there is growing concern that the competitive project funding system favors relatively safe, conventional projects at the expense of risky, novel research. It is important to assess different funding models in order to design a a better funding system for science. This paper empirically tests for differences in the novelty of funded outputs between internal block and competitive project funded papers, in the setting of Japan where both funding models play a significant role. Combining survey data from a large sample of research projects in Japan and bibliometric information about the publications produced from these projects, we find that projects funded by competitive funds on average have a higher novelty compared with those funded by internal block funds. However, such positive effects only hold for researchers with high academic status, such as senior and male researchers. On the contrary, compared with internal block funding, competitive project funding has a much stronger detrimental effect on novelty for low status scientists (junior and female researchers). The findings suggest that the competitive project selection procedure is less receptive to novel ideas from researchers with low academic status and therefore discourages their novel research. These findings can serve as a warning about potential bias in competitive funding allocation procedures and suggest the importance of secured stable funding for allowing researchers with low status to pursue their novel ideas.

Paper short abstract:

The paper studies how the national evaluation system of public research in the Netherlands developed in the context of science policy and university governance. It puts a specific focus on the use and relevance of bibliometric indicators as a tool for measuring scientific quality and excellence.

Paper long abstract:

This paper studies the evolution of the national system of assessment of public research in the Netherlands in the context of Dutch science policy and university governance. It puts a specific focus on the use and relevance of bibliometric indicators as a tool for measuring scientific quality and excellence in this evaluation practice and relates this to the development of the Centre for Science and Technology Studies (CWTS) as an expert organization and main provider of bibliometric analyses in the context of the Dutch evaluation protocol.

From an experimental use of bibliometric methods during the establishment of evaluation procedures for selected disciplines in the seventies and eighties, bibliometric methods became an optional part of the standardized protocols of the Association of Dutch Universities (VSNU) at the discretion of the disciplinary committees in the nineties. Currently they remain a voluntary indicator for demonstrating scientific quality in the Standard Evaluation Protocol (SEP) if deemed necessary by the research unit or university boards. The declining relevance of bibliometrics at the national assessments is compensated by their growing role in annual administrative-managerial routines at universities.

The study employs document analyses, a detailed literature review and expert interviews to determine the share of bibliometric analyses in VSNU and SEP evaluation reports from 1993 to 2014 and to investigate motivations for using bibliometrics.

The paper contributes to an understanding how the partial definition of excellence through bibliometric indicators is transformed from a national, disciplinary matter into a problem of technical decisions at the university management level.

Paper short abstract:

Interviewing staff from an Irish scientific funding agency, I explore how they use formal and informal mechanisms to foster engagement between universities and industry, to define the meaning of “excellent science” in Ireland and to push university researchers to act and think like entrepreneurs.

Paper long abstract:

University-industry research centers are a growing global trend with the goal of bridging the gap between academic knowledge creation and industry commercialization. The university and industry, as social institutions, have different norms, cultures, goals and informal and formal rules, yet States often see their convergence as an opportunity for innovation and economic growth. So how does a State coordinate engagement between these two, often contradictory, institutions? Using in-depth interviews with staff from an Irish scientific funding agency, I explore how this agency foster university and industry collaboration in Ireland. I argue that the agency often leverages its research funding to define the meaning of "excellent science" and to shape and reshape the role university researchers play in knowledge production, pushing them towards acting and thinking of themselves as "entrepreneurs," and at the same time, preserving their status as "disinterested" knowledge producers. Formally, the agency uses the grant application process to push university researchers to think about and talk about their scientific research in terms of its economic impact. To maintain the claim to producing "excellent" and "disinterested" science, the agency uses an "unbiased" international peer review panel to evaluate the quality of the science, university researchers produce. Informally, agency staff engage with university researchers by coaching, guiding, and advising in order to reshape and modify their ambitions and bring them more in line with industries commercial goals. I argue that it is these formal and informal mechanisms that ultimately shapes the kinds of science that emerges in Ireland.

Paper short abstract:

Coherent narrative stories based on 14 narrative biographical interviews with young researchers vividly describe disciplinary, institutional differences related to excellence polices, and the evolution of such policies that is uneven and produces different results in different settings.

Paper long abstract:

This paper intended for presentation uses 14 narrative biographical interviews with young researchers from different disciplines and fields. The research was framed with the goal of gathering and analyzing narratives related to career advancement and personal development in different institutional and disciplinary settings within Croatian academic system. The main intention related to research question was, through researching the most vulnerable population within the system - younger researchers, to understand how changes in science and higher education polices influence disciplinary and institutional cultures. Interviews are guided in narrative mode, gathering oral history related to professional development of respondents ranging from their high school interests to present date competencies and future aspirations. Respondents have spontaneously told their life-history stories of their research development by highlighting events important in their own view, describing their institutional and disciplinary environment(s) and therefore giving us a context at different stages of their careers. Narrative data that consists of actions, events, and happenings is analysed for every interview with the focus on discourse related to excellence and polices of advancement in career in order to produce coherent stories. These coherent stories vividly describe disciplinary and institutional differences related to excellence polices, but also the evolution of such policies that is uneven and produces different results in different settings.

Paper short abstract:

A focus on scientific impact seems to be predominant both in the rhetoric and the evaluating practices of excellence, neglecting the social relevance of research. Using a qualitative approach, we compare researchers’ opinions and experiences on these matters in the Spanish context.

Paper long abstract:

Science policies reward researchers on the basis of a widespread notion of excellence associated with scientific impact. This reward system has decisive consequences for shaping scientists' behaviour by orienting their research goals. However, much of scientists' personal sense of achievement and social recognition comes from their perceived contribution to societal needs and challenges. In this sense, a research gap exists regarding the imbalance between concepts of impact and relevance shaping the discourse and the assessment of scientific excellence. In this study we trace the meanings that researchers give to their achievement of both scientific impact and social relevance and how these perceptions influence their research agendas and knowledge transfer practices.

Our methodological approach focuses on qualitative content analysis drawing on a set of interviews conducted with Spanish researchers between October 2015 and March 2016. The interviewees (10 men and 10 women) work at public research organisations (universities and research centres) in a wide range of scientific disciplines. We compare researchers' views, perspectives and experiences in the field of Social Sciences, Humanities and Arts with those in the STEM disciplines. Taking as a point of departure researchers' own selection of key results in their scientific careers, we analyse their perceptions on three elements: scientific impact, social relevance and excellent science.

Our preliminary findings suggest that scientists' differentiated narratives about scientific excellence influence whether they embrace interaction with non-academic actors within their research agendas. Understanding the main factors and dynamics influencing researchers' thinking about excellence in science has potentially important policy implications.

Paper short abstract:

Focusing on the national programme of Centres of Excellence in scientific research, this paper outlines the patterns and distribution of the centre of excellence programme in Finland.

Paper long abstract:

The recognition of excellence in science takes many forms in the scientific community, often through epistemic community- based recognition, but also through active peer reviews focusing on the existing institutions and individuals (e.g. Lamont, 2009). However, one specific form of the recognition of excellence is tied to the institutional funding mechanisms of science. The science funding institutions (research councils, innovation and science agencies, national academies) have in alignment with epistemic communities developed funding instruments for the Universities and institutions to reach the international excellence in science. Many of these instruments travel and get copied transnationally and domesticated nationally in slightly differing ways. But how does the recognition of the excellence become settled and defined at the national level when the funding is mainly at stake at the national level? What types of settled performance criteria are being used for different disciplines?

Focusing on the national programme of Centres of Excellence in scientific research, this paper first outlines the patterns and distribution of the centre of excellence programme in Finland. Second, the paper relates to and discusses the empirical data in relation to different disciplinary fields, and the concept of excellence in relation to the intensifying economic discourse, with references to 'return of investment and 'impact'. Overall, the paper suggests that the centre of excellence discourse works around the concept of basic research, which for social sciences may prove to be problematic when set against the criteria of impact of science currently often required from CoEs.

Paper short abstract:

This paper aims to gain insights into the possible implications of excellence policies on the degree of differentiation in a science system and the effects of these policies on research practices and epistemic content. To this end, we analyze Dutch funding data and interviews with 16 Dutch research groups.

Paper long abstract:

Many science systems witness the rise of excellence policies, funding instruments that selectively support high-performing and high-potential individuals or organizations, in order to increase differentiation.

Scholars have expressed a number of concerns arguing that excellence policies may create a system where the 'winner takes all', that the focus on competition may hinder cooperation, or that the quality of education and broader impact recede into the background.

To date, the systematic understanding of the effects of excellence policies on scientific practices is limited. The aim of this paper is to fill this knowledge gap by analyzing the interplay between excellence funding and academic research practices in The Netherlands.

To this end we undertake three empirical endeavors. First, we analyze the patterns of excellence funding; to what degree can we observe a concentration of funding at a limited number of individuals, research groups or research areas? Second, we report on case studies of four research groups that benefit strongly from excellence funding; we show how the multiple values of excellence funding for these research groups amplifies the Matthew effect. Third, we compare the research practices of groups benefiting from excellence funding with the research practices of twelve groups not benefiting from excellence funding; do these groups construct alternative notions of excellence?

By synthesizing various types of empirical data we gain insight into the implications of excellence policies on the degree of differentiation and their effects on academic research practices and epistemic content.

Paper short abstract:

We try to assess the co-production and promotion of research indicators at the science-policy interface, with particular focus on the Research Excellence in Science & Technology indicator developed by the European Commission. We offer ways to re-imagine such indicators from a STS perspective.

Paper long abstract:

In the advent of the evidence-based society, indicators are increasingly called upon to inform policymakers, not least in the domain of research and innovation policymaking (Barre, 2010).

Few studies, however, have scrutinized how such indicators come about in practice, leaving much of the back-stage work on indicators-for-policy invisible (Nowotny, 2007). The aim of this paper is to assess the co-production of research indicators at the science-policy interface and to make visible the processes through which research indicators are produced, promoted, and used by research policymakers.

We particularly focus on the Research Excellence in Science & Technology (RES& T) indicator offered and used by the European Commission. We have been actively involved in the design and construction of this indicator (Hardeman et al., 2013) and, hence, are in a unique position to critically engage in assessing its co-production process. In so doing, our goal is to come up with ways for re-imagining the measurement of research excellence for policy purposes.

Reviewing the literature (Sorensen et al., 2015), we outline the tensions inherent to the disputes on defining and measuring research excellence for policy purposes.

Second, using both our own reflections on the construction of the RES& T indicator and interviews with relevant EU policymakers, national policymakers, and other stakeholders, we assess how and to what extent the RES& T indicator has been respectively produced and used in practice. Finally, we identify gaps in the current RES& T indicator as well as opportunities for measuring research excellence to inform research policies.

Paper short abstract:

My paper proposes to map the different ways to frame the notion of excellence in four institutions. It will compare how “good research” is valued in different settings, pointing out four conventions that convey distinct technologies and moral principles to govern research practices.

Paper long abstract:

During a stay at Lancaster University, I was struck to discover how the institutional branding relied on the rhetoric of excellence. Several flags scattered over the campus are displaying statements such as: "our physicists helped discover the Higgs boson particle", "Lancaster University is ranked among the top 10 universities in the UK", "our volcanologists made the first observation of a rare type of lava", etc. These quotes convey technologies such as rankings, but also moral and epistemological aspects regarding the role and status of scientific knowledge. This fostered my reflection regarding the fieldwork I made in different research institutions located in Belgium: while academic excellence is not at the core of their branding per se, it is indeed a key issue for many actors concerned with research governance.

My paper proposes to map the different ways to frame the notion of excellence in four institutions: two large biotech research centres and two universities in Flanders and Wallonia. It relies on 25 semi-structured interviews with tenured academics involved in their institution's research management. The paper will compare how "good research" is valued in different settings: what counts when one evaluates the "excellence" of a scientific production/career? To which technologies, indicators and to which "moral principles" are interviewees referring to when they discuss excellence? How do they engage in criticizing certain framings and praise others? Eligible for the rhetoric or comparative panel, my contribution will present four distinct institutional conventions that govern research practices and sorts the good scientist from the bad.

Paper short abstract:

In recent years, concerns have been raised about excellence policies in science leading to unfruitful competition and the accumulation of scientific credits in the hands of few. But what is the role of excellence policies within the entire range of science policies? Policymaker talks back.

Paper long abstract:

Qualifying science as excellent plays a major role in (inter)national science policies and funding schemes, and in institutional policies such as thematic profiling, talent development programmes and investments. As an STS-educated senior policymaker in a public research institute, I will analyse practical, theoretical and normative aspects of the notion of excellence in science policies. Rather than putting excellence on the suspect bench, I will argue that a focus on excellence provides precious room for manoeuvre for research that displays the "wow!"-factor. Alternatives will be discussed as well.

Public funding agencies are under growing pressure to maintain and legitimize their budgets in terms of economic and societal impact. As a consequence, funding for research with no manifest out-look towards return on investment is vulnerable. In this context, how does the classification of science as 'excellent' provide for a boundary object? And how does this shape funding schemes, selection procedures, and the content of research proposals?

A focus on excellence drives the policies of research institutes as well. In such professional organisations, next to policymakers, scientists themselves might be highly involved in developing policy instruments. This steering paradigm of 'the professional in the lead' fits well with the notion of fostering excellence. The involvement of scientists in institutional policy making is however no warranty to prevent marginalization of certain groups of researchers and scientific disciplines.

Finally, I will explore what normative issues are at stake in, on the one hand, excellence policies and on the other, in questioning excellence policies in science.

Paper short abstract:

The scientific system works optimally when excellence as potential or performance is rewarded. We discuss the role of research funding in this system, current trends in research funding and their potential effect on funding policies’ ability to turn potential into excellence.

Paper long abstract:

The scientific system works optimally when excellence, both as a potential and as a performance, is recognized and rewarded. Since researchers obtain funding in exchange for a "promise" of future accomplishment, it can be seen as being an instrumental form of reward which recognizes excellence as potential. However, the boundaries between these two forms of excellence are blurred when past performance is used as an indicator of the potential for future performance. For example, grant proposals are not evaluated using blind reviews and funding decisions are based not only one the quality of the research proposals but also on the quality of the researchers, which is to a large extent assessed through their past accomplishments. It has been observed that because of the growing scarcity of research funds and the trend towards larger grants, research funding is increasingly playing an honorific rather than instrumental role in the scientific system. Indeed, if we consider an optimal funding policy as one that allows both to sustain researchers' achieved excellence as well to realize researchers' potential for excellence, current trends in funding policy might be favoring the first goal at the expense of the second. Using data on the research funding and the scientific output of researchers in Quebec, we discuss how the effect of current policy trends on the performance of researchers, and how such changes might affect funding policies ability to play an instrumental role in the pursuit of scientific excellence.

Paper short abstract:

While science policy often focusses on individual excellence, majority of research today is collaborative. This paper suggests concept of ‘collaborative excellence’, explores practices and governance mechanisms supporting it, and discusses implications for science policy, funding and evaluation.

Paper long abstract:

'If one wants to work in a collaborative research project, he should be willing to share his knowledge and should be willing to understand that the progress of research might be risky in a sense that it comes from the team, it does not come from individual. […] I personally do not care if a discovery in [our collaborative project] is done by my group or is done by other group. […] What interests me is success of [our collaborative project], the success of our community and that we do something new and discover something.'

This quote from a group leader and collaborative project leader in nanosciences demonstrates how much scientific excellence is a team effort. While majority of research in STEM disciplines is collaborative (Bozeman and Boardman 2014), policies for evaluating and funding excellent science still tend to favor individual excellence, for example, assuming that co-authored publications should have a lead author who should receive more credit or that science prizes should be allocated to individuals. Against this background, this paper suggests a concept of 'collaborative excellence'. On the basis of extensive research on scientific collaborations (Ulnicane 2015), it explores how high quality research is generated, organized and evaluated within networks. Questions addressed include: What science practices, policy mechanisms and governance arrangements support and hamper 'collaborative excellence'? What is relationship between 'individual' and 'collaborative' excellence? What indicators could be useful in evaluating 'collaborative excellence'? Implications for science policy, funding and evaluation will be discussed.