This session explores expanding metascience beyond quantitative methods, addressing data access challenges, and using qualitative, behavioral, and content analysis techniques to deepen understanding of scientific processes and team science.
Long Abstract
As metascientists, our work is shaped by the data and methods we use - enabling us to develop theories and test hypotheses, yet also imposing limitations. To deepen our understanding, it is crucial to go beyond purely quantitative approaches. This session begins with a paper addressing real-world challenges in accessing high-quality data and explores solutions through technology and collaboration. It then features three contributions demonstrating how qualitative and behavioral methods can enrich our insights into the science of team science across various contexts. The final two papers introduce innovative techniques for analyzing publication content—such as data tables and text—to uncover insights that metadata alone cannot reveal.
Drawing from Elsevier-supported initiatives, we'll explore novel indicators and emerging datasets addressing components of the research ecosystem beyond publications, discuss challenges in accessing high-quality data, and highlight opportunities offered by LLMs and collaborative solutions.
Long abstract
Metascience aims to understand and explain phenomena across the research enterprise by developing tools for analysing the inputs, throughputs, outputs, and outcomes of research. This foundational work has paved the way for the emerging field of ‘applied metascience,’ which employs theoretical models and applied techniques - including indicators - for evaluative purposes. While significant strides have been made in understanding research outputs (such as publications and the citations that connect them), the field is now more intentionally addressing other components of the research ecosystem.
A persistent challenge is the availability of high-quality, comprehensive data. Even when data are available, a further challenge is making it useful through disambiguation, harmonisation, and linking to other data (including those held locally by individuals and institutions).
In this presentation, we will share examples from Elsevier-supported initiatives that leverage new and emerging datasets to advance metascience and its applications in evaluation. While these examples are seldom fully realized solutions, they serve as proof-of-concept illustrations of the ‘art of the possible’. We will focus on examples relating to the process in which research is done and the broader societal and economic impacts it has, and emphasize the complex challenges driven by real-world demands.
We will conclude by underscoring the necessity of collaborative efforts to tackle these ‘wicked problems’. By working together, we can address some of the most pressing issues facing the research community today.
Metascience can be more than quantitative metrics and models. This paper makes the case for the use of qualitative data, focusing on research-on-research as a ‘meta’-approach. We propose longitudinal qualitative analysis as a novel approach to R-o-R and interdisciplinary team science.
Long abstract
Metascience is frequently associated with statistical analysis, metrics and quantitative modelling. Complementary to this however, qualitative data provides different and additional insights which can increase the explanatory potential metascience and therefore should not be overlooked. This paper makes the case for the use of qualitative data, more specifically the insights it can offer as a component of an overall metascience, research-on-research method. Drawing on qualitative data collected through three rounds of interviews and workshops with research consortium members during a six-year research programme, this paper shares learnings regarding the operationalisation of effective inter- and transdisciplinary (ITD) team working.
TRUUD, Tackling Root Causes Upstream of Unhealthy Urban Development, is a six-year research consortium bringing together over 40 academics across six UK universities. Focusing on the prevention of non-communicable diseases through urban development, the consortium draws on expertise across several disciplines and stakeholders. This has provided an opportunity to increase the understanding of team working in the context of effective and efficient ITD research. Three rounds of interviews and workshops with TRUUD members were carried out 1-1.5 years apart to inform the research-on-research findings discussed in this paper. Theoretically our approach is anchored in team science and science of team science literature. The analytical framework combines systems thinking with longitudinal qualitative analysis to observe development of behaviour in the context of ITD in a complex team setting. We show the value of this approach and encourage others in the metascience community to engage with and build on qualitative research-on-research.
Big team science enables robust research with low-prevalence populations by mitigating small sample sizes and heterogeneity. We present a multinational study on cognition and intellectual disability to highlight challenges and solutions, demonstrating how careful planning drives meaningful progress.
Long abstract
Big team science provides valuable opportunities for metascience by enabling the curation of large databases, standardising research protocols, improving data reliability, and facilitating long-term data collection for future analyses. Such large-scale collaborations are especially valuable for research involving low-prevalence populations such as clinical groups, individuals with disabilities or individuals with specific characteristics. Small sample sizes and heterogeneity remain significant limitations in conducting studies with such populations, often limiting the generalisability, reliability and metanalytical syntheses of findings. As a consequence, some fields fail to make efficient progress despite many papers being published every year.
In this presentation, I will propose big team science as a solution promoting significant progress in research involving low-prevalence populations. I will reflect on key recruitment, methodological and logistical challenges encountered in our large-scale study on cognition and intellectual disability conducted across 30 research groups in 20 countries. Addressing these challenges required continuous collaboration, adaptability, and engagement with local research teams to balance standardisation with cultural and contextual relevance.
I will also share practical solutions and lessons learned from our project, emphasising the importance of proactive problem-solving in large-scale, multinational studies involving low-prevalence populations. By openly discussing the issues we aim to demonstrate that, while challenging, conducting robust, inclusive, and diverse studies is achievable with careful planning. We hope that this will encourage more researchers working with low-prevalence populations to adopt the big team science approach in the future and, as a result, push their fields towards meaningful progress.
This mixed-methods study used the Behaviour Change Wheel develop an intervention to improve collaborative research leadership among researchers. Findings identified influences on leadership such as communication skills, role models and competitiveness. The process evaluation is ongoing.
Long abstract
Rationale and Aims
Improving research culture is a growing priority in universities, with collaboration and leadership being key areas of focus. As these are forms of human behaviour, behavioural science frameworks can be used to understand drivers of collaborative leadership, to inform design of improvement strategies. This study applied the Behaviour Change Wheel (BCW) approach to develop an intervention to improve collaborative research leadership, therefore contribute to enhancing research culture at UCL.
Methods
This mixed-methods study involved intervention development whereby we conducted 31 semi-structured interviews with research-active staff to explore influences on collaborative leadership. Topic guides were based on the Capability-Opportunity-Motivation-Behaviour (COM-B) model of behaviour. Interview transcripts were analysed using deductive framework and inductive thematic analysis. Identified influences were mapped to potential intervention strategies using the BCW and Behaviour Change Technique (BCT) taxonomy.
Findings
Influences on collaborative leadership were identified across all COM-B domains. Enablers included having the knowledge of what collaborative leadership entails and how to perform this, having compatible goals, and the professional network(s). Barriers included managing the mental workload, having other (higher) priorities, and outcomes being driven by the competitive environment rather than team science. Examples of interventions proposed included skills coaching, 360 feedback with colleagues, and goal setting in reflective journals.
Conclusions
This study demonstrates how behavioural science frameworks can be used to develop an intervention aimed at enhancing research culture, particularly in collaborative leadership. The process evaluation is ongoing to assess fidelity of delivery and acceptability of the intervention.
We developed a tool that automatically extracts statistical values from tables (DORIS), resulting in a sample of 578,132 statistical tests from economics. We analyze (1) the prevalence of statistical reporting errors and (2) the effects of data and code availability policies.
Long abstract
We developed a tool that automatically extracts statistical values from tables (DORIS), resulting in a sample of 578,132 statistical tests from the top 50 economics journals (1998-2016). We analyze (1) the prevalence of statistical reporting errors that occur if the eye-catcher depicting the level of statistical significance is inconsistent with the reported statistical values. 14.88% of the articles have at least one strong reporting error in the main tests and there is a bias toward statistical significance. We also analyze (2) the effects of data and code availability policies on characteristics of published articles. We use a staggered difference-in-differences design, showing that these policies result in slightly less emphasis on statistical significance, more rigor in reporting, fewer reporting errors in control variables, but not in more citations.
DORIS can be used in the review process to improve article quality and to generate data at a large scale for future meta-research.
The identification of geographical entities in research outputs is important for science monitoring/evaluation. To address this demand, we developed a tool that finds locations in text, matches them to their unique spatial footprint and classifies them according to their role (e.g. object of study).
Long abstract
Accessing the semantic contents of research outputs is essential for science monitoring and evaluation. One type of information found in research documents is their geographical scope: what places are studied or where does the research take place? Geotagging, the task of identifying and disambiguating geographical mentions in text, not only captures specific geographical points as relevant for a research activity, but it also allows for the definition of new indicators.
To address this demand, we built a multilingual system (which will be available open-source) that performs the following tasks: (1) Identification of geographical entities in scholarly texts, (2) Toponym resolution, (3) Contextual role classification (e.g. object of study or impacted location).
For (1), we created a new NER dataset focusing exclusively on geographical entities by combining existing datasets from multiple sources. With it we finetuned a multilingual LLM (CAT, DE, EN, ES, FR, IT) to perform geographical NER (following Zekun et al 2023). For (2), the OpenStreetMap API is called and the mentions are matched with their most likely unique spatial footprint. For (3), we trained a classifier with a dataset of geographical mentions in R&I contexts manually labelled with their role, based on the following taxonomy:
-Object of study
-Location of research
-Impacted location
-Contextual/other
This last step allows for higher analytical granularity and the elimination of possible noise.
In our talk, we will present a case study on a territorial ecosystem (analysing their publications, and R&I projects) demonstrating the impact of the tool for metascience and research mapping.
Short Abstract
This session explores expanding metascience beyond quantitative methods, addressing data access challenges, and using qualitative, behavioral, and content analysis techniques to deepen understanding of scientific processes and team science.
Long Abstract
As metascientists, our work is shaped by the data and methods we use - enabling us to develop theories and test hypotheses, yet also imposing limitations. To deepen our understanding, it is crucial to go beyond purely quantitative approaches. This session begins with a paper addressing real-world challenges in accessing high-quality data and explores solutions through technology and collaboration. It then features three contributions demonstrating how qualitative and behavioral methods can enrich our insights into the science of team science across various contexts. The final two papers introduce innovative techniques for analyzing publication content—such as data tables and text—to uncover insights that metadata alone cannot reveal.
Accepted papers
Session 1 Tuesday 1 July, 2025, -