New methods and data infrastructures are expanding our capacity to generate collective intelligence about the effectiveness, agility and direction of research systems. In this session, we will hear examples of these approaches from the OECD, Overton and European Research Infrastructure for Life Science Data (ELIXIR) — as well as from researchers in the UK, US, Czechia and Slovakia.
Long Abstract
This session will provide horizon scan of rapidly developing tools and infrastructures with the potential to overcome barriers at the research-policymaking interface. Daniela Valenzuela will describe how the STIP Compass database collects and structures evidence and leverages AI to facilitate impactful analyses of global Science, Technology and Innovation Policy landscapes, while Basil Mahfouz will report on an innovative machine learning pipeline for detecting policy-relevant science to enable systematic analysis of the science-policy interface at scale that can reveal how scientific evidence flows into policymaking.
New ways to help researcher and policymakers engage more productively despite ever increasing research volumes will be the focus of Ceire Wincott’s discussion of Overton Engage, a new tool that applies structured data, natural language processing (NLP), and automated policy-matching algorithms, to aggregate policy engagement opportunities into a single, accessible platform to connect researchers with policy needs.
Two further talks will explore new methods for mining information and value from existing metaresearch and research resources - work that is to be of great relevance to policymakers. Benjamin Simsa will report on the accuracy and cost-effectiveness of using Large Language Models for automating the extraction of information from scientific papers (ranging from verbatim information such as effect sizes or preregistration status to subjective inferences) to accelerate metaresearch. And Peter Maccallum will discuss work at ELIXIR, the European Research Infrastructure for life science data, on an Impact Toolkit to evaluate the global significance and value of its resources, finding that its impact goes well beyond citation and usage.
The STIP Compass database enables metascience research on government support for STI in 60+ countries. It is based on a recurring government survey and enables exploration of thousands of STI policy instruments. Both data collection and presentation are augmented with latest AI technology.
Long abstract
STIP Compass is an online interactive data repository that gathers qualitative and quantitative data on national science, technology, and innovation (STI) policies. Maintained by the OECD and European Commission, it integrates structured data of STI policies with statistics, academic research, and other evidence sources to examine how science is governed, funded, and supported. Combining traditional data collection with methodologically innovative AI-assisted techniques, it enables large-scale studies of STI policy design and institutional frameworks shaping science and innovation.
As STI policies influence the development and operation of research systems, STI policy datasets are a fundamental metascience resource. The primary STIP Compass database originates from a biennial expert survey among STI policymakers that currently provides structured data on over 7,900 policy initiatives from 60 countries and the EU. This database is structured around six core policy areas: Governance, Public research, Innovation in firms, Knowledge exchange and co-creation, Human resources for research and innovation, and Research and innovation for society.
At the technology frontier of AI-enhanced STI policy data collection, STIP Compass employs LLMs to streamline data collection, improve curation, and synthesise policy trends. These AI-assisted methods allow for more efficient data processing and unlock new ways of exploring policy landscapes across countries.
By providing structured insights into policy trajectories, institutional frameworks, and funding mechanisms, STIP Compass offers a unique empirical foundation for analysing STI policy measures that shape research systems worldwide. This paper presents how STIP Compass collects and structures evidence and leverages novel methodologies to explore global STI policy landscapes.
A machine learning pipeline that detects policy-relevant science and enables the first systematic analysis of the science-policy interface at scale. This talk reveals key insights in how scientific evidence flows into policymaking, with implications for improving evidence-based governance.
Long abstract
The gap between scientific evidence and policymaking remains a critical challenge for effective governance. Current research on the science-policy interface has largely relied on analysing how governments cite scientific papers, overlooking potentially valuable research that fails to reach policymakers.
We present a novel machine learning pipeline that detects policy-relevant research via paper abstracts. This enables the first systematic, large-scale analysis of both used and overlooked policy-relevant science. When applied to large bibliometric databases, our model reveals previously hidden patterns in how scientific evidence flows into policymaking, including institutional factors and author characteristics that influence policy uptake. These insights inspire new tools for improving the transmission of scientific evidence into policy, ultimately supporting more effective evidence-based governance.
ELIXIR, the European Research Infrastructure for life science data, will present the challenges in measuring the global significance and value of the resources it represents and its approach to impact and metrics beyond usage and citation.
Long abstract
The European Strategy Forum on Research Infrastructures has worked for decades to build a rich ecosystem of services for scientific equipment, sample repositories, data and software. In this model, the European Commission and member states co-fund a managed portfolio of capabilities for researchers; but in a world of limited resources, choices must be made. How do we assess the value of Research Infrastructure investments? And what is the value of data in that landscape?
ELIXIR is the European Research Infrastructure for life science data, and represents the major biological data resources of 21 Member states. Among the hundreds of data resources identified, we have created a formal process to identify the most critical, the Core Data Resources. We will set out the reasoning behind that selection, the experience of curating the list and measuring its use by researchers, and the challenges of communicating their significance. One key theme is that impact goes far beyond citation and usage; to this end ELIXIR has developed an Impact Toolkit to help its resources self-assess and maximise their relevance, and a range of metrics and indicators relevant to other distributed infrastructures and large scale scientific endeavours with impact beyond the literature.
We test the feasibility of Large Language Models for automating data extraction in metaresearch. Results show that these models achieve high accuracy in extracting a wide range of metascientific variables at a fraction of the cost of manual coding, with frontier models nearing human-level accuracy.
Long abstract
The manual data extraction in metaresearch is often a tedious, time-consuming, and error-prone process. In this paper, we investigate whether the current generation of Large Language Models (LLMs) can be used to extract accurate information from scientific papers. Across the metaresearch literature, these usually range from extracting verbatim information (e.g., the number of participants in a study, effect sizes, or whether the study is preregistered) to making subjective inferences.
Using a publicly available dataset (Blanchard et al., 2023) containing a wide range of meta-scientific variables from 34 network psychometrics papers, we tested six LLMs (Claude 3.5 Sonnet, Claude 3 Opus, Claude 3 Haiku, GPT 4o, GPT 4o mini, o1-preview). We used the API for extracting the variables from the documents automatically. This automated pipeline allows batch processing of research papers. As such, it represents a more efficient and scaleable way to extract metascientific data, compared to using the default chat interface.
Our results point to a high accuracy and high potential of LLMs for metascientific data extraction. The accuracy of the respective models ranged from 76 % to 87 %, and most models were able to convey uncertainty in the more contentious cases.
We provide a comparison of accuracy and cost-effectiveness of the individual models and discuss the characteristics of variables that are (non)suitable for automatic coding. Furthermore, we describe some of the common pitfalls and best practices of automatised LLM data extraction. The proposed procedure can decrease the time and costs associated with conducting metaresearch by orders of magnitude.
Researchers often struggle to find opportunities to engage with policy, while policymakers face hurdles in accessing relevant expertise. This research presents a centralised repository that aggregates policy engagement opportunities into a single, accessible platform.
Long abstract
The integration of research evidence into policymaking is critical for addressing global challenges, yet academic-policy engagement remains fragmented and inefficient. Researchers often struggle to find relevant opportunities to contribute their expertise, while policymakers face difficulties identifying and accessing the right research. These barriers slow the translation of knowledge into policy action and limit the role of metascience in improving institutional decision-making.
This research presents a metascientific intervention: a centralised repository that aggregates policy engagement opportunities—such as calls for evidence, advisory roles, and collaborative research initiatives—into a single, accessible platform. By reducing logistical barriers and leveraging AI-driven discovery tools, this system strengthens institutional research cultures, fosters inclusive participation, and accelerates knowledge exchange between academia and policy.
By applying structured data, natural language processing (NLP), and automated policy-matching algorithms, this work demonstrates how digital infrastructure can systematically connect researchers with policy needs. Overton Engage provides a scalable solution for identifying relevant research in real time. Integrating this approach into institutional processes enables governments, funders, and universities to streamline knowledge mobilisation, making policy engagement more efficient, transparent, and inclusive.
This work highlights the potential for long-term alliances between academia, policymakers, and infrastructure providers to sustain and scale these innovations. By embedding data-driven engagement mechanisms within research institutions, this approach offers a model for optimising academic-policy interactions and ensuring that research systems continuously evolve to meet society’s most pressing challenges.
Short Abstract
New methods and data infrastructures are expanding our capacity to generate collective intelligence about the effectiveness, agility and direction of research systems. In this session, we will hear examples of these approaches from the OECD, Overton and European Research Infrastructure for Life Science Data (ELIXIR) — as well as from researchers in the UK, US, Czechia and Slovakia.
Long Abstract
This session will provide horizon scan of rapidly developing tools and infrastructures with the potential to overcome barriers at the research-policymaking interface. Daniela Valenzuela will describe how the STIP Compass database collects and structures evidence and leverages AI to facilitate impactful analyses of global Science, Technology and Innovation Policy landscapes, while Basil Mahfouz will report on an innovative machine learning pipeline for detecting policy-relevant science to enable systematic analysis of the science-policy interface at scale that can reveal how scientific evidence flows into policymaking.
New ways to help researcher and policymakers engage more productively despite ever increasing research volumes will be the focus of Ceire Wincott’s discussion of Overton Engage, a new tool that applies structured data, natural language processing (NLP), and automated policy-matching algorithms, to aggregate policy engagement opportunities into a single, accessible platform to connect researchers with policy needs.
Two further talks will explore new methods for mining information and value from existing metaresearch and research resources - work that is to be of great relevance to policymakers. Benjamin Simsa will report on the accuracy and cost-effectiveness of using Large Language Models for automating the extraction of information from scientific papers (ranging from verbatim information such as effect sizes or preregistration status to subjective inferences) to accelerate metaresearch. And Peter Maccallum will discuss work at ELIXIR, the European Research Infrastructure for life science data, on an Impact Toolkit to evaluate the global significance and value of its resources, finding that its impact goes well beyond citation and usage.
Accepted papers
Session 1 Tuesday 1 July, 2025, -