Click the star to add/remove an item to/from your individual schedule.
You need to be logged in to avail of this functionality.
Log in
- Convenors:
-
Stephanie Meirmans
(Amsterdam UMC, University of Amsterdam)
Jeannette Pols (University of Amsterdam)
Maarten Derksen (University of Groningen)
Jonna Brenninkmeijer (Amsterdam UMC)
Send message to Convenors
- Format:
- Combined Format Open Panel
Short Abstract:
What is the epistemic value of replication studies and open science? Worldwide, open science and reproducibility are proclaimed as ways to improve science. The panel explores the ‘goods’ that emerge from such efforts, and we take stock of how STS scholars react to and engage in them.
Long Abstract:
Worldwide, open science and replication are proclaimed as ways to improve science. Networks are created to strategically shift the attention of researchers, funders, publishers, and universities to raise transparency and reproducibility. Many STS scholars are involved in such efforts, be it as drivers, supporters, interviewers/observers, advisors, or as critical voices to (parts of) the endeavour.
In this panel, we will bring together STS experiences and reflections regarding such reproducibility and open science efforts and networks. The panel explores the ‘goods’ that emerge from desires and attempts to replicate studies and the implications of various ways of making science more open. What might be unintended side effects? We take stock of how STS scholars react to and engage in such efforts. How and when are STS scholars critical and where do they engage with the goods of replication studies and open science? With what epistemic understandings do they engage, and what strategies do they employ to change academic practices ‘for the better’? What is this ‘better’? And what could they learn from such efforts?
The panel explores how STS insights and approaches – such as valuation studies and everyday empirical ethics – might help to understand the issues at stake and intervene. We critically explore the diversity of values pertaining to reproducibility and open science.
We invite participants to submit proposals about these topics. We particularly invite for presentations in the form of storytelling about their personal experiences and engagement in these issues, but we would also welcome either more traditional presentation formats or other, more creative, forms of proposals.
Accepted contributions:
Session 1Stephanie Meirmans (Amsterdam UMC, University of Amsterdam)
Short abstract:
Should we as science scholars intervene in the reform movement, and if so, how can we do so effectively? Part of the problem is that STS scholars and reformers may live in their respective bubbles. Here, I tell some stories on ways of alignment work that can help to overcome some problems.
Long abstract:
Should STS scholars and philosophers intervene in the reform movement, and if so, how? In practice, such intervention attempts can be unsuccessful because science scholars and reformers typically live within their own bubbles or worlds. Recent STS work (see special issue in Science and Technology Studies, 2023, 4) has focused on ‘alignment work’, which I suggest is a good way to think around some of the problems that can emerge with such interventions: after all, one needs to efficiently translate between the two epistemic cultures to have the intended impact. And indeed, a misunderstanding can go both ways: how can we also understand them better? How to interact in effective ways across the two bubbles?
Here, I will share some stories of ways in which I and my colleagues have done forms of alignment work in our current project on ‘replications in action’. In our ethnographic work, we closely interact with advocates of the open science reform movement. We achieved some success with impact and collaborations by the close and prolonged interaction with replicators, by asking critical questions in interviews and field work, by hosting workshops with replicators and reformers, by co-writing papers with replicators, and by presenting talks in the open science community. In connection with this, I want to emphasize that perhaps the biggest impact we as science scholars can have on the reform movement is not by producing scientific output that only reaches our own community, but by making efforts to reach out.
Megan Finn (American University) Amelia Acker (The University of Texas at Austin) Yubing Tian Thomas Struett (American University) Sarika Sharma (Middlebury College)
Short abstract:
New laws passed say that scientists have to included data management plans in funding applications to the National Science Foundation in the United States. This paper uncovers the reasons for this law and how data management plans can be interpreted by science and technology studies scholars.
Long abstract:
The United States’ science data policy has landed in a weird place. Data management plans are now required to ensure “research reproducibility and replicability” on National Science Foundation grants according to new legislative code. There are several reasons given in various administrative rules for more access to and transparency around research data including equity, research replicability, projecting the superiority of American science, ensuring the return on investment to taxpayers, incentivizing better research data management, legitimizing a future for data science and data-driven research, and enhancing the quality of science. But there is little evidence that data management plans achieve these ends. This paper asks, how did the United States end up with “data management plans” as a requirement? And, what do data management plans tell us about the future of science data? We examine the history of data management plans. Then, we draw from our corpus of nearly 1000 data management plans from projects funded by the National Science Foundation-funded and delineate different approaches to reading data management plans: as scientific furniture, as instruments for accessing funding, as evidence of the neoliberalization of science, as a process document for scientific knowledge production and institutional coordination, as a fantasy document, as an indicator of the future, aligning temporalities, and as part of the institutionalization of data-oriented science. By sharing findings about the purposes and premises of planning for data management we argue they are an important vehicle for understanding the political project of open science in the United States.
Célya Gruson Daniel (Inno3)
Short abstract:
This proposition is a reflexive feedback on the integration of reproducibility in a mixed methods study called “Adapting Open Science”. It combines quantitative rigor, with data and code sharing, and qualitative depth using grounded theory analysis and explanation within “reasoning traceability”.
Long abstract:
This contribution to the field of Science and Technology Studies (STS) draws from the “Adapting Open Science” study, conducted by a multidisciplinary team in France, emphasizing a mixed-methods approach that marries quantitative reproducibility with “reasoning traceability” for qualitative depth. The quantitative phase involves an external verification of the results of a survey based on the meticulous data sharing in the French national repository (recherche.data.gouv) then the production of a Jupyter notebooks documented following literate programming principles and shared in Gitlab. The qualitative phase (interviews, focus group), guided by Pierre Paillé’s 'grounded theory analysis', highlights the importance of “reasoning traceability” providing an alternative to “traditional reproducibility” proposed for quantitative data analysis. Both phases benefit from the editorialization of a website, aimed at making the methodology and findings more accessible. This intervention critically evaluates reproducibility in research, providing a nuanced perspective on open science practices.
The study and all the process are documented in the dedicated website (in French) : https://declinerso.pubpub.org/. A summary of the study is available in English : https://hal-lara.archives-ouvertes.fr/hal-03800470v1
Sven Ulpts (Aarhus University) Jesper Wiborg Schneider (Aarhus University)
Short abstract:
We provide a framework for the assessment of the relevance and feasibility of different types of 'reproducibility' (replication) in diverse research contexts to support 'reproducibility' where it is relevant and prevent inappropriate demands for it where it is irrelevant.
Long abstract:
We propose a framework for the assessment of the relevance and feasibility of different kinds of ‘reproducibility’ (replication) in the realm of epistemic diversity. ‘Reproducibility’ is clouded in an immense conceptual confusion within and across disciplines and research approaches. Furthermore, ‘reproducibility’ captures a variety of different practices and functions. Therefore, the first component of our framework helps stakeholders identify crucial parts of ‘reproducibility’ that enable the assessment of its relevance and feasibility in a specific research situation. Importantly, the diversity in knowledge production and justification across the research landscape implies a diversity in research characteristics that affect the appropriateness of ‘reproducibility’. We distinguish between the relevance based on epistemic considerations and the feasibility based on practical aspects of the specific research situation. The relevance is assessed based on whether the function of ‘reproducibility’ (e.g., generalizability, reliability, or validity) makes sense for the kind of knowledge sought and whether it fits into the research quality (trustworthiness) criteria and practices of the type of research. The feasibility is assessed based on, nature of the subject of investigation, the availability of the resources that the research practices depend on, and on the degree of uncertainty that is associated with specific research practices. One of our aims is to enable stakeholders to identify cases where they could put their limited resources to facilitate ‘reproducibility’ when it is relevant but not (yet) feasible. However, our main aim is to prevent the application of a ‘reproducibility’ demand where it does not belong.
Maarten Derksen (University of Groningen)
Short abstract:
We report on a replication of the social studies of replication conducted in STS in the 1970s and 1980s. We conducted an ethnographic study of 16 replication studies in social science to see whether the conclusions that Collins and others drew from their studies still hold.
Long abstract:
The idea that direct replication is essential to verify or falsify scientific claims is well established among scientists and metascientists. However, in STS it has been criticized. Since the 1970s sociologists, philosophers, historians and anthropologists of science have studied replication in practice and have shown that replication is considerably more complicated than it seems, and does not offer an objective criterion regarding the truth or falsity of scientific claims. Harry Collins’s work on replication in the 1970s and 1980s has been particularly influential. In this paper we report on a replication of these social studies of replication. Whereas Collins and colleagues focused mainly on natural science, we extend their work to social science, where replication is increasingly common but also controversial. We conducted an ethnographic study of 16 replication studies in social science to learn more about the motivations of replication researchers and the practical and methodological problems they encounter. Why do these replication researchers conduct replication studies, and how are their aspirations of reproducibility affected by the problems they encounter in conducting their replication studies? Do the conclusions that Collins and others drew from their studies still hold?