This poster will introduce Atlas of Assessment, a website mapping national research evaluation systems worldwide. By simplifying complex differences, it enables researchers and policymakers to compare, learn, and understand diverse national research assessment practices via its interactive platform.
Paper Abstract
The number of national research assessment systems has expanded dramatically across many countries in recent years, although there is no single formula. In fact, designs and rationales vary considerably, from performance based funding systems to feedback-oriented advisory procedures, from systems relying on qualitative peer review to quantitative bibliometrics methods, and from those focusing on evaluating the performance of individual researchers to entire universities or disciplines, the sheer diversity is disorientating. Given this situation, what can be learnt from comparing diverse national systems and how can this be done in a legible, accessible way to support mutual learning?
Step forward the Atlas of Assessment, a state-of-the-art web resource presenting expertly curated data on national research assessment systems from around the world. Based on a cutting-edge typology that simplifies, categorizes and compares multiple dimensions of these complex systems, the Atlas will serve as a publicly available resource enabling policymakers, institutional leaders and researchers to browse countries, highlight regional trends, and learn about what others are doing.
The Atlas will be launched in the weeks running up to Metascience 2025.
The aim of this poster will be to introduce meta-researchers to this new resource, walk them through its main features, provide navigation instructions, suggest potential use cases, and provide a QR link to the website.
The Atlas of Assessment is a non-commercial product, publicly available to all. It is a co-production effort between meta-researchers in the Research on Research Institute's AGORRA project and expert policymakers and funding administrators from 13 countries (and counting).
GRIOS aims to advance Open Science by synthesizing existing evidence to guide policies, identify barriers to adoption and highlight knowledge gaps. The poster will present GRIOS and share results from its first study assessing the coverage of Open Science topics in the scientific literature.
Paper Abstract
GRIOS, the Global Research Initiative on Open Science, is dedicated to deepening our understanding of Open Science practices and fostering their widespread adoption among governments, funding agencies and research institutions.
Emerging from G7 work on Research on Research and Open Science, and inspired by the IPCC's role in synthesizing climate science for policymakers, GRIOS will employ evidence synthesis methods to assess the benefits and limitations of Open Science practices.
GRIOS will aim to identify the obstacles causing uneven uptake of Open Science policies across countries and organisations. By addressing these challenges, the initiative aims to support funders, institutions and policymakers in refining their strategies and maximizing the impact of Open Science policies.
GRIOS will also aim to identify critical knowledge gaps and federate the global research-on-Open Science communities around a common research agenda.
The first study commissioned by GRIOS aims to better understand the Open Science research landscape. It has developed a methodology to measure research on the topic using an open data source, namely OpenAlex. Based on a set of specific keywords and journals across four languages (English, French, Portuguese and Spanish), the analysis provides an overview of the main Open Science topics studied, the countries that are active, the study design used, the disciplines in which those papers are published, and the open access availability of papers. The study also presents challenges in using open data sources for the measurement of science, such as the heterogeneous data quality across document types, languages, and publishers.
Replicating 81 articles in top economics journals, I show that 36-63% of estimates defending the average null result fail lenient equivalence tests, implying high Type II error rates. I provide equivalence testing guidelines and software commands to help researchers make more credible null claims.
Paper Abstract
Equivalence testing can provide statistically significant evidence that economic relationships are practically negligible. I demonstrate its necessity in a large-scale reanalysis of estimates defending 135 null claims made in 81 recent articles from top economics journals. 36-63% of estimates defending the average null claim fail lenient equivalence tests. In a prediction platform survey, researchers accurately predict that equivalence testing failure rates will significantly exceed levels which they deem acceptable. Obtaining equivalence testing failure rates that these researchers deem acceptable requires arguing that nearly 75% of published estimates in economics are practically equal to zero. These results imply that Type II error rates are unacceptably high throughout economics, and that many null findings in economics reflect low power rather than truly negligible relationships. I provide economists with guidelines and commands in Stata and R for conducting credible equivalence testing and practical significance testing in future research.
This talk explores open science in developing countries—examining its potential to democratize research, the challenges of limited infrastructure and funding, and strategies to foster sustainable, inclusive innovation through global collaboration and tailored policy solutions.
Paper Abstract
This talk examines how open science can transform research ecosystems in developing countries. Open science—through principles of transparency, collaboration, and rapid dissemination of knowledge—offers a unique opportunity to democratize research and accelerate innovation. However, implementing open science practices in resource-constrained environments presents significant challenges, including limited digital infrastructure, funding shortages, and policy gaps.
Drawing on case studies and emerging initiatives, this presentation will highlight both the promise and pitfalls of open science in these regions. It will explore how access to global knowledge networks, international funding opportunities, and collaborative research models can empower local scientists and institutions. Simultaneously, the talk will address critical obstacles and propose actionable strategies to overcome them. These include fostering local capacity building, encouraging public-private partnerships, and developing policies that support sustainable open research practices.
By presenting a balanced view of both opportunities and challenges, the session aims to spark a dialogue on how stakeholders—from policymakers to academic institutions—can work together to create an equitable global research landscape. Attendees will leave with a clearer understanding of how to leverage open science for inclusive growth and sustainable innovation in developing countries.
The Scientific Software funding line of the German foundation Klaus Tschira Stiftung tackles challenges in research software engineering through systemic support and participatory formats such as Sandpits and Pitches. This talk will showcase innovative funding approaches and key lessons learned.
Paper Abstract
The Scientific Software funding line of the German foundation Klaus Tschira Stiftung exemplifies how innovative funding instruments can address systemic challenges in research software engineering, a critical yet under-supported component of modern science. Despite its central role in enabling complex research, scientific software development often lacks sustainable structures, leading to inefficiencies, reproducibility issues, and a loss of long-term value.
This program leverages metascientific principles to enhance the quality, sustainability, and impact of research software by focusing on structural innovations. It fosters systemic improvements, such as creating permanent roles for Research Software Engineers, building institutional capacities, and promoting best practices in software development.
A hallmark of the program is its participatory approach to funding. Using a Sandpit format, the Foundation engages researchers, funders, and stakeholders in collaborative workshops to co-create project ideas, integrating diverse perspectives and fostering new alliances. Additionally, the use of Pitches as an alternative to traditional written proposals prioritizes creativity, reduces administrative burdens, and ensures a human-centered approach to funding.
By embedding principles of experimentation, collaboration, and learning, the Scientific Software program directly aligns with metascience’s broader goals: improving the processes, infrastructures, and cultures that underpin scientific research. This talk will explore how these innovative funding methods can serve as a model for supporting research infrastructures, demonstrating how metascience can drive practical change. It will also highlight lessons learned and discuss how such approaches can be scaled to advance institutional support for research software engineering globally.
We conduct a series of “many-teams” analyses on a variety of government impact evaluations to understand analysis-dependence in government research. By partnering with social researchers from the UK Cabinet Office, the findings from our project will directly inform research practice in government.
Paper Abstract
How reliable are the impact evaluations conducted by government? Impact evaluations – studies which measure the effects of policy interventions – play a central role in government decision-making, and UK government departments spend hundreds of millions of pounds on conducting and commissioning evaluation activities each year.
Recent metascience research demonstrates that many quantitative research findings are highly analysis-dependent: researchers investigating the same research question, using the same data, often make radically different analytical choices, and draw highly contrasting conclusions. If the estimated effects of a policy intervention vary substantially with analysts’ choices, we cannot be sure that the information from any single analysis that feeds into policymaking is correct. Understanding how sensitive government impact evaluations are to analysis decisions is therefore crucial for improving the robustness of evidence-based policymaking.
Our project, which is funded through a UKRI Metascience Research Grant, aims to generate new evidence on the robustness of quantitative research in government and provide actionable guidance on ameliorating analysis dependence for government researchers. First, we build on recent metascientific work and conduct a series of “many-teams” analyses to re-evaluate the estimates of a broad variety of existing government impact evaluations. Second, our project unites an academic team with government social researchers in the UK Cabinet Office, which enables us to translate our findings into new guidance on mitigating analysis-dependence which will be incorporated into the Magenta Book, a key resource for all evaluators across UK government (due to be published in 2025).
The paper explores the rise of impact services at UK universities in connection to the establishment of impact as one of the evaluation criteria in REF. The survey conducted at 156 institutions evidences the creation of new roles, positions and institutional practices.
Paper Abstract
The establishment of research impact as an evaluation criterion in the British Research Excellence Framework brought about notable changes in the way academic work is understood and organized. While changes to the academic environment, academic ethos and academic discourse have been somewhat explored in existing studies (Chubb & Watermeyer, 2017; Watermeyer, 2014; Wróblewska, 2021), this paper attempts an overview of the rise of impact services. We focus on new roles (impact officer) and positions (impact champion) emerging in the area of support for impact generation as well as institutional practices (prizes for impact). This article builds on a survey conducted among 156 institutions in the UK. We advance a hypothesis regarding the emergence of an impact infrastructure around the new academic value – ‘research impact’. The study will be of interest to scholars of academic culture and governance as well as to practitioners.
While in principal a paper proposal, we are happy to provide a poster in addition.
Chubb, J., & Watermeyer, R. (2017). Artifice or integrity in the marketization of research impact? Investigating the moral economy of (pathways to) impact statements within research funding proposals in the UK and Australia. Studies in Higher Education, 42(12), 2360–2372.
Watermeyer, R. (2014). Issues in the articulation of ‘impact’: The responses of UK academics to ‘impact’ as a new measure of research assessment. Studies in Higher Education, 39(2), 359–377.
Wróblewska, M. N. (2021). Research impact evaluation and academic discourse. Humanities and Social Sciences Communications, 8(1), Article 1.
We investigated if and how qualified statisticians were consulted by human research ethics committees in Australia. We encountered a stark variance in attitudes and practice, with statisticians being called both “critical” and “not needed for 99.9% of applications”.
Paper Abstract
Currently, much medical research is wasted due to errors in the study design or analysis. Errors can be reduced by consulting qualified statisticians with expertise in a wide range of study designs and methods. Ethical review is an ideal stage to consider study design, however, we do not know how many statisticians are providing their expertise to ethics committees.
We approached all human research ethics committees in Australia with questions on their access to qualified statisticians. Sixty percent of committees had access to a qualified statistician, either as a full committee member or as a non-member who could be consulted when needed, but this result dropped to 35% after accounting for statistical qualifications. Many committees rely on “highly numerate” researchers instead of qualified statisticians, as they viewed research experience and advanced statistical training as equivalent. Some committees did not feel the need for a statistician, as they believed it was the institution's job to ensure good study design. Other committees believed that researchers could be trusted to submit robust study designs. There was a common belief that statistical review only applied to selected study designs, and that “simple” or “small” studies did not need scrutiny. We estimated that around 5,200 applications per year were seen by a qualified statistician across all committees combined, whilst around 8,000 were not.
We found a stark variance in views and practices across Australia. Current ethics review processes risk approving studies that at best waste resources and at worst cause harms due to flawed evidence.
This paper uses the case of handedness in neurodevelopmental disorders to illustrate how databases from existing meta-analyses can be leveraged for second-order meta-analyses. This involves updating and reanalyzing previously published meta-analyses in identical analysis pipelines.
Paper Abstract
This paper explores handedness in neurodevelopmental disorders to demonstrate how databases from existing systematic reviews and meta-analyses can be utilized for second-order meta-analyses. To this end, we reviewed and updated previously published meta-analyses concerning hand preference in mental and neurodevelopmental disorders, aiming to identify overarching patterns and estimate the influence of potential moderators independent of diagnosis. A total of 402 datasets encompassing 202,434 individuals were analyzed. Our findings indicate that atypical hand preference, including non-right, left, and mixed preferences, is significantly more prevalent in clinical samples compared to controls, with odds ratios (OR) for non-right of OR = 1.46, 95% CI = [1.35;1.59]; for left of OR = 1.34,95%, CI = [1.22;1.48], and for mixed of OR = 1.63, 95% CI = [1.38;1.93]. Notably, disorders such as schizophrenia exhibit particularly high rates of atypical hand preference (non-right OR: 1.50, 95% CI = [1.32;1.70]; left OR: 1.37, 95% CI = [1.17;1.61]; mixed OR: 1.70, 95% CI = [1.19;2.44]). Moderator analyses revealed that neurodevelopmental conditions, non-neurodevelopmental conditions with early onset, and conditions with language-related symptoms are all linked to higher rates of atypical hand preference. This research indicates that the relationship between handedness and psychopathology is best understood from a transdiagnostic, developmental, and symptom-focused perspective. From a methodological standpoint, our study underscores the potential of second-order meta-analysis to enhance trust in scientific findings.
To enact an ecosystem change, TIER2 has built a community of funders to foster reproducible research practices in funded projects. Through three co-creation workshops, they have drafted the Reproducibility Promotion Plan (RPP) with actionable recommendations and best practice examples.
Paper Abstract
Multiple stakeholders across research and innovation argue that we are experiencing a credibility revolution. Reproducibility and Open Science are essential for ensuring integrity and quality in research. In response, various stakeholder groups are advocating for changes at the structural, cultural, and infrastructural levels of the scientific ecosystem. Research funders play a crucial role in this ecosystem and are therefore vital levers of change through their funding calls and requirements researchers have to comply with. They can catalyze improvement and behavioral change among researchers through funding conditions, policies, tools, and infrastructures. Currently, funders are not only prioritizing innovation but also advocating for sustainability in research by building infrastructures necessary for responsible research practices.
In this context, supported by the EU funded project TIER2— Enhancing Trust, Integrity, and Efficiency in Research, has built a community of funders with expert knowledge on implementing tools to increase reproducibility. Through three co-creation workshops, funders have drafted the Reproducibility Promotion Plan for Funders (RPP). This plan offers recommendations across three key areas of funding work: evaluation and monitoring, policy and definitions, and incentives. The RPP provides actionable recommendations and best practice examples that funders and funding institutions can adapt to meet their specific needs. The RPP aims to support funders in promoting reproducibility, Open Science, and quality in the research they fund. Presently, we are piloting the RPP with international funding organizations of various sizes and are seeking community feedback using a survey to further refine and validate the RPP.
This study explores attitudes to and experiences with preregistration in Swiss animal researchers, highlighting negative attitudes, low awareness, and perceived barriers such as bureaucracy, time costs, and low flexibility. These findings offer guidance in view of facilitating preregistration.
Paper Abstract
Background and aims: Preregistration has long been established as standard practice in clinical research and is increasingly taken up in other fields. However, it remains uncommon in animal research. This study aimed to:
1. evaluate experiences with study preregistration in animal researchers in Switzerland;
2. describe their attitudes, subjective norms, perceived behavioral control, intentions, and motivations, and obstacles regarding preregistration;
3. identify associations for these outcomes;
4. identify the perceived facilitators and barriers of preregistration; and
5. derive suggestions for interventions to facilitate preregistration.
Methods: A cross-sectional online survey was conducted among all study directors of ongoing animal experiments in Switzerland.
Results: Of the 1,386 study directors, 418 (30% return rate; 41% female; age M = 46.9, SD = 9.2) completed the survey. Among them, 39% had never heard of preregistration, and only 10% had preregistered studies before participating in the survey. On average, participants reported rather negative attitudes towards preregistration, negative subjective norms, relatively low perceived behavioral control, intention, and motivation, as well as high perceived obstacles regarding preregistration. Bureaucracy (78%), time costs (71%), and low flexibility (66%) were the most common perceived barriers to preregistration. However, participants with less research experience and those with preregistration experience expressed less negative views about preregistration.
Discussion: These findings provide detailed insights into the barriers and facilitators of preregistration in animal research. They thus offer guidance on future interventions aimed at facilitating preregistration of animal research in Switzerland and beyond.
Co-produced by academics and publishers (incl. CUP, Cell Press, EMBO Press, Taylor & Francis, GigaScience Press, OUP, PLOS, Springer Nature), the Handbook assists in-house editorial staff to operationalise a set of checks fostering good sharing data practices (https://publishers.fairassist.org).
Paper Abstract
A workshop with representatives of major publishers (https://doi.org/10.17605/OSF.IO/TGUXZ, 2023) indicated that strengthening their journals’ data policies and training their in-house editorial staff were among the key priorities to improve the availability of data underpinning publications, and foster good practices for sharing data, ultimately advancing open research.
Fast forward two years, the Editorial Reference Handbook (https://publishers.fairassist.org) informs and assists journals to operationalise a set of checks necessary to make data more findable, accessible, interoperable and reusable. The Handbook complements existing work (e.g., to improve data policy and analyse the impact on journals of introducing Data Availability Statements) and fills a gap, because no common guidance existed on the practical implementation of these checks across a complex publishing workflow and the variety of individuals and teams who handle a manuscript.
Beside this practical product, the Handbook is also a socio-technical pilot to improve the culture by facilitating the practice and leading by example, influencing and informing other publishers and journals. The use of the Handbook is being piloted by a number of journals, and the ongoing intervention (set to end this summer) aims to document what may need to change or improve to successfully implement these checks in terms of in-house capability (e.g., needing more knowledge about how to run them), opportunity (e.g., needing support to apply them), and motivation (e.g., needing to prioritise them).
The work is part of the European TIER2 project (No 101094817).
We evaluated eleven confirmatory multi-laboratory projects to assess preclinical research robustness. Our analysis revealed enhanced rigor as the project progressed from exploratory to confirmatory stage. We examined effect sizes and significance to compare single- and multi-lab studies.
Paper Abstract
Many promising preclinical research findings fail to translate into clinical practice, as numerous interventions ultimately fail during the costly clinical trial phase. One proposed strategy to improve evidence generation along the preclinical trajectory is introducing multi-laboratory (multi-lab) confirmatory studies with increasd rigor and transparency. To investigate their potential benefit, we assessed eleven projects that conducted confirmatory research in a rigorous multi-lab set-up. We accompanied the projects during their four-year funding period and provided methodological support. Through this, we have generated a unique database that allowed us to analyze protocols and primary data from exploratory phase, pre-registration, and confirmatory stage. We evaluated the reliability and validity of the projects through a guided self-assessment of robustness, aiming to identify potential improvements from the exploratory to the confirmatory phase. Overall, most projects showed enhanced research rigor over time. Internal validity improved through the implementation of blinding and randomization, while external validity was strengthened by incorporating diverse disease models, including both sexes, and implementing a multi-lab study design. However, projects also reported set-backs within the multi-lab setup, for example differences in animal welfare regulations resulting in severe delays. We conducted a meta-analysis of each project's confirmation success, evaluating effect sizes and statistical significance. In conclusion, our analysis of a convenience sample offers a nuanced perspective on the translational relevance and potential pitfalls of preclinical confirmatory multi-laboratory studies.
Methods Hub is a platform enhancing reproducibility in Computational Social Science by structured method sharing, standardized submissions, and integrated checklists. It fosters transparency and community-driven validation, ensuring the reusability and reliability of computational research methods.
Paper Abstract
Reproducibility remains a key challenge in Computational Social Science, with many researchers struggling to implement best practices. Methods Hub is an initiative designed to improve the reusability and reproducibility of computational methods by providing structured submissions, standardized reproducibility checklists, and community-driven validation. Integrated within the broader goals of TIER2, Methods Hub offers a niche platform where researchers can share, review, and refine computational methods applied to social science use cases.
Our recent survey of 180 participants revealed significant gaps in reproducibility practices: 40% of researchers are unfamiliar with key reproducibility standards such as FAIR and TOP, while 67% have never used checklists to ensure the reproducibility of their work. Additionally, 24% of respondents rarely or never find others' research reproducible or reusable based on the available documentation, with only 14% consistently achieving these goals.
By integrating reproducibility checklists and fostering community feedback, Methods Hub aims to bridge these gaps, ensuring transparency and reliability in computational research. This initiative aligns with broader efforts to standardize and improve research practices in the Social Sciences.
OpenSAFELY is a research platform designed with transparency, reproducibility and security in mind. It is currently deployed to the UK’s electronic health records. Come and speak to us about the platform’s philosophy, design features, and how it could be used in your work.
Paper Abstract
Electronic health records (EHR) are among the UK's most valued and sensitive data assets. To maintain credibility, research in EHR must be transparent and reproducible, and to preserve trust, it also must adhere to the highest security and privacy standards. OpenSAFELY is an analytic software platform developed to meet these needs. It enables verified researchers to run analytical code transparently and reproducibly against millions of linked health records, and returns aggregated and disclosure-proof outputs, preserving the privacy of individuals’ data.
Features that support OpenSAFELY’s reproducibility include: standardised data preparation workflows, implementation of the same computational environment across all users, a universal query language to generate analysis-ready datasets, and a library of reproducible actions. For transparency, researchers can only run analyses by sharing their code on GitHub and all analyses conducted on the secure server are logged in public. To preserve privacy, researchers do not have access to individual-level data. The analysis is prepared against the dummy data, and can only be released with disclosure controls after review by output checkers.
Our poster presents the OpenSAFELY design and how we are working to change the EHR research lanscape. We invite you to learn more about the platform's features and philosophy, engage with the OpenSAFELY researchers and developers, ask 1:1 questions, and learn how OpenSAFELY could be useful in your work.
While promoting open research to ensure transparency, researchers should remain reflective throughout the research project when engaging with AI tools and research should be conducted in a way that follows a balanced approach, where advantages and risks around GenAI are taken into consideration.
Paper Abstract
While promoting open research practices to ensure transparency, accessibility and rigour, researchers should also remain reflective throughout the entire cycle of their research project when engaging with AI tools. Research should be conducted in a way that follows a balanced approach, where we consider both: advantages and risks that are associated with GenAI. We are in a transition stage in the academic environment where we are adjusting policies or putting new policies in place to address artificial intelligence. As librarians, we think about new ways of supporting researchers so that they can follow a responsible approach when inviting generative AI tools into their research practice. To enable a sustainable research ecosystem, we are addressing those new ways of working with data. GenAI tools should be used effectively but also ethically and transparently which fits into the principle around open research practices. In this poster I have presented different aspects linked to each stage of the research project ( planning stage, active stage and dissemination stage). In the descriptive parts I have highlighted the key questions that should be considered when following a reflective approach while introducing GenAI tools into the research process. Is Ai permitted in the study? How should we use GenAI? And what would be the comprehensive way to communicate the use of GenAI upon dissemination of the research findings.
What is measured in surveys investigating the uptake of Open Science? While most surveys include 'open access' in the operationalization of Open Science, practices such as 'open peer review', 'open educational resources', and 'public engagement' are only sporadically measured.
Paper Abstract
What is measured in surveys investigating the uptake of Open Science? While most surveys include 'open access' in the operationalization of Open Science, practices such as 'open peer review', 'open educational resources', and 'public engagement' are only sporadically measured. This poster presents results from a literature study on the operationalization of Open Science practices in survey research.
The literature study was conducted at the Open Science Retreat 2024 (https://openscienceretreat.eu/). The team consisted of Lotte van Burgsteden, Tanya van Goch, Bogdana Huma, Anne Marie Meijer, Iris Smal and Hanne Oberman. Together, the team collected, reviewed, and analyzed existing research into open science practices. As a team, we developed an interactive overview of open science surveys, which may be used e.g. to reuse questionnaire items on different open science practices.
The main categories we encountered were 'Open Access', 'Pre-printing', 'Open Peer-review', 'Open Data', 'Open Code', 'Pre-registration', 'Open Educational Resources', and 'Public Engagement'. A dashboard for visualizing the findings is available via https://hanneoberman.shinyapps.io/os-surveys/. All of our study materials are openly available, with the literature list on Zotero (https://www.zotero.org/groups/5464665/osr24nl), code on GitHub (https://github.com/oscutrecht/OpenScienceSurveys), and a persistent identifier on Zenodo (https://doi.org/10.5281/zenodo.10932820). We also inventoried the psychological constructs measured by the various surveys and identified seven constructs: awareness, attitudes, behavior, intention, barriers, benefits and conditions.
Evaluating research performance is crucial for assessing academic contributions and impact. Traditional bibliometric methods often fail to capture complex patterns in citation networks and author collaborations. Generative Adversarial Networks (GANs) will enhance research metrics evaluation easily.
Paper Abstract
Assessing research performance is essential for understanding academic contributions, citation impact, and researcher productivity. Traditional bibliometric approaches rely on citation counts, h-index, and impact factors, but they often struggle to capture the complex, dynamic nature of scholarly influence. This study introduces a novel framework leveraging Generative Adversarial Networks (GANs) to enhance research metrics evaluation. The proposed GAN-based model synthesizes realistic citation patterns, detects anomalies in research impact, and predicts future trends with greater accuracy. By training on extensive bibliometric datasets, the model effectively learns underlying citation structures and academic network behaviors. Experimental validation demonstrates that the GAN-based approach improves ranking precision, detects citation manipulation, and provides a more holistic evaluation of research performance than conventional methods. Furthermore, this model facilitates early identification of influential works and emerging research domains, offering valuable insights for institutions, funding agencies, and policymakers. The proposed method represents a significant advancement in data-driven research assessment, moving beyond static metrics toward adaptive, AI-powered evaluation techniques. Integrating deep learning with bibliometric analysis provides a more robust and comprehensive framework for analyzing the scholarly impact, ensuring fairer and more accurate assessments of academic contributions. The GAN-based model generates synthetic citation patterns and predicts future research trends, enabling a more accurate and dynamic assessment of scholarly impact. Experimental results demonstrate improved precision in ranking researchers and publications compared to conventional methods. This framework offers a data-driven approach to refining research evaluation systems.
This paper will explore the use of research evidence in the decision-making process(es) affecting research cultures within UK HEIs and consider how the use of metascience in institutional decisions can be improved to enhance research culture, rigour and transparency.
Paper Abstract
Initiatives to understand and improve institutional research cultures, which in turn promote research excellence, should be informed by insights and evidence from robust research. However, recent research suggests that evidence-informed decision-making on issues affecting research cultures is not widely embedded within UK HEIs, despite recognition that there is a body of work that could be drawn from. The limited application of metascience to understand and improve institutional research cultures puts fair and inclusive working environments at risk. Limited understanding of the change required for a healthy and thriving research environment as well as a lack of robust evidence used to justify decisions can lead to resistance and/or disengagement with change and adversely affect the organisation's culture.
This paper will explore the use of research evidence in the decision-making process affecting research cultures within UK HEIs. The research draws on interviews from around 80 participants working within six organisations across three categories of research intensity, including representatives from both academic and professional services staff. The paper will begin by outlining the existing decision-making process(es) with HEIs, commenting on the similarities, differences, relationships and interplay between decision-making at the institutional and departmental level, before reflecting on the use of evidence and the extent to which metascience is applied. It will conclude by considering how the use of metascience in institutional decisions can be improved to enhance research culture, rigour and transparency.
Open data-sharing is increasingly prioritised by publishers, funders, and institutions. We assess 555 bioscience and 114 circadian mental health papers using manual and automated methods. Our findings reveal trends, challenges, and key factors driving improvements in FAIRness and reproducibility.
Paper Abstract
As scientific research increasingly values outputs beyond traditional publications—such as datasets, software, and code—the need to assess the openness and FAIRness of shared data has become more urgent. Funders, publishers, and institutions are prioritising data-sharing, yet its implementation varies widely across disciplines.
This talk examines data-sharing practices in biosciences at the University of Edinburgh from 2014 to 2023, analysing 555 research papers across biotechnology, regenerative medicine, infectious diseases, and non-communicable diseases. We extend our analysis to 114 publications in UK MRC Circadian Mental Health Network to identify domain-specific trends.
Using a manual scoring system, we assessed data completeness, reusability, accessibility, and licensing. Our findings show a significant improvement in bioscience data-sharing: by 2023, 45% of studies shared all relevant data, compared to just 7% in 2014. These figures stand in contrast to Hamilton et al. (2019), which reported 19% data-sharing in cancer research by 2019. However, circadian mental health research lags, with only 8% of studies sharing data in 2023. Genomic datasets were more frequently shared than image or human subject data, and data availability statements (DAS) and preprint sharing strongly correlated with higher data-sharing practices.
We also evaluated the automated tool ODDPub (Open Data Detection in Publications) (Riede et al. 2020), which demonstrated high specificity in identifying studies without shared data and improved sensitivity when better documentation was present.
These findings underscore both progress and persistent gaps in open data-sharing. They highlight the need for clearer policies, improved infrastructure, and automated tools to support reproducibility and FAIRness across disciplines.
We assessed reliability and measurement reporting in 77 Many Labs replications and related original articles. We evaluated the impact on replication informativeness and success, and advise minimum measurement reporting standards and informed replication choice.
Paper Abstract
In the wake of the “replication crisis”, many scientific reform initiatives have focused on promoting replication. However, if the measurement in the original study lacks reliability and construct validity, the findings of a replication study will not be informative. Recent studies have found issues with the measurement in both original and replication studies. However, the impact these issues have on the informativeness of replications remains unclear. We assessed the reliability and measurement reporting practices of 77 measures within 56 Many Labs replications and related original articles (Ebersole et al., 2016, 2020; Klein et al., 2014, 2018), as well as their impact on the replication success. First, we observed that reliability was not sufficient in each context for several measures. Second, measures were rarely accompanied with reliability and validity evidence. Third, questionable measurement reporting practices - but not reliability - were associated with lower replicability in our sample. These results corroborate existing findings that construct validity in published research is often unknown, which may reduce the informativeness of replication research. We discuss how academic stakeholders can improve of measurement through consistent use of minimum standards for construct validity, and by including reported measurement information as part of the decision process in deciding what to replicate.
Scientists increasingly publish in Open Access (OA) journals. This study compared the statistical quality of psychotherapy research in OA vs. subscription journals. OA journals showed larger samples, higher statistical power, and smaller effect sizes, supporting their role in robust research.
Paper Abstract
Scientists increasingly publish in Open Access (OA) journals, which offer free access to scientific literature and are considered more transparent and rigorous in their editorial and peer-review processes. While Open Science promotes methodological and statistical quality, no study has examined whether this principle is reflected in OA versus subscription-based journals in psychological sciences. This study addresses this gap by comparing the statistical quality of publications assessing the efficacy of psychotherapies for depression in adults.
We retrieved 467 effects from 357 articles in the Metapsy database (Miguel et al., 2022), including 199 effects from 167 OA journal articles and 268 effects from 190 subscription journal articles. We compared observed effect sizes, sample sizes, and statistical power across publication types.
Effects in OA journals exhibited (i) smaller effect sizes, (ii) larger sample sizes, and (iii) higher statistical power compared to subscription journals. Median statistical power was below the recommended 80% threshold for small (d = 0.2) and medium (d = 0.5) effect sizes in both publication types. However, for large effect sizes, OA journal articles had a median statistical power above 80%, while subscription journal articles remained below this threshold.
Our findings suggest that OA journals publish studies with higher statistical quality, likely due to their greater transparency and rigorous standards.
The COReS project develops a framework to integrate systematic review into preclinical research through education, infrastructure, and community building. Piloted across Germany, COReS enhances research synthesis and incorporates evidence-based decision-making into preclinical research.
Paper Abstract
The Communities for Open Research Synthesis (COReS) project develops a framework to initiate systemic change in how preclinical research is translated into improved human health outcomes. Systematic review and meta-analysis are research synthesis tools that act as an evidence-based bridge to clarify existing knowledge, assess data reliability, and identify research gaps. We employ a three-pillar approach to integrate preclinical systematic reviews into the research pipeline.
Education: We enhance awareness and capacity through live workshops and a suite of freely available eLearning modules covering theall systematic review and meta-analysis steps. Our “train-the-trainer” programme scales this education by equipping trainers to teach this methodology at their institutions.
Infrastructure: To carry out systematic reviews, appropriate infrastructure and support is required. We build on existing software, the Systematic Review Facility (SyRF), improving user experience, documentation, and helpdesk services. New features are developed in collaboration with diverse user groups, and we integrate novel automation tools to reduce resource demands and streamline the systematic review process.
Community building: Forging communities to bridge the disconnect between primary research and evidence synthesis is instrumental in promoting an integrated research pipeline. Our digital hub brings together resources, tools, and support in one place. Our open online community forum fosters collaboration, standard sharing, and joint project development. Regular events spark new connections and dissemination of approaches.
Five German partner institutions are piloting this community blueprint to ensure adaptability across institutions and biomedical fields. By fostering interdisciplinary networks, COReS strengthens evidence-based decision-making and advances translational research.
PREreview leverages preprints to promote open and equitable peer review, challenging the traditional model. Through community building, training, and technology, we are leading meaningful change. This poster highlights 360° feedback workflows, our Champions Program, and Live Reviews in action.
Paper Abstract
Preprints offer the opportunity to challenge and reform the way researchers engage with the evaluation of each other’s work. Traditional peer review is an opaque, biased, and antiquated process that for too long has remained in the hands of a few, for-profit publishers that have used it to effectively control the fate of knowledge production and dissemination.
PREreview harnesses preprints to create a more open and equitable alternative to peer review. Our approach integrates community building, training and mentorship, and human-centered technology to foster constructive and transparent scholarly dialogue.
This poster will showcase the core pillars of our work through community-driven stories. Specifically, we will highlight:
a) 360° feedback workflows—how PREreview enables open, trackable review cycles where authors request feedback, reviewers provide input, and authors respond.
b) Championing change—a firsthand account from our Champions Program, illustrating how mentorship and leadership training empower researchers to foster open peer review in their communities.
c) Live Reviews in action—insights from years of facilitating collaborative, real-time review sessions that promote inclusivity, dialogue, and shared best practices.
By centering community engagement and equity, PREreview is reimagining peer review as a transparent, participatory process that serves researchers at all career stages.
From 1978 till end of 2024, 67 studies analysed differences between preprinted or submitted versions vs peer reviewed journal versions. Our meta-analyses show that while some aspects change, conclusions rarely do (in 2% of cases). We are launching a living review website to showcase all results.
Paper Abstract
Previous research has indicated a knowledge gap on changes brought on by peer review and journal publishing. We are conducting a living evidence synthesis of studies that analysed differences between preprinted or submitted versions vs peer reviewed journal versions. We identified 67 studies published from 1978 till the end of 2024, of which 33 (49%) analysed changes between preprint and journal versions, 26 (39%) between submitted and published versions, and 10 (15%) between rejected versions and those later published in other journals. All but 3 studies looked at different sets of outcomes. Furthermore, 45 (67%) analysed changes manually, 14 (21%) used computational methods, and 8 combined the two (12%). The median number of analysed version-pairs was 109 (IQR 48 to 388). Narrative synthesis indicated that studies reported high similarity between version-pairs, with meta-analyses showing highest frequency of changes in title, authorship, conflicts of interest, and numerical results, with rare impact on study conclusions (i.e. 2% of analyzed pairs, 95%CI 1 to 4). However, with almost 7 million articles published per year, the combined sample size covering almost half a century of research on this topic is likely inadequate for proper results generalizability, especially as no data points exist for many fields. We have therefore approached publishers and editors to greatly increase data availability that would allow larger samples to be compared. We are also launching a live tracker website that we plan to present at the conference.
The EC-funded PathOS project explores the academic, societal, and economic impacts of Open Science. This talk summarises our findings incl. evidence reviews, case studies, new indicators, cost-benefit analyses, as well as impact assessment challenges and recommendations for the future.
Paper Abstract
Open Science is increasingly mainstream, with growing adoption of practices globally. While systems and data sources for monitoring uptake are maturing, research or systems to gauge long-term, real-world impacts remains scattered and uneven, with large gaps in data and knowledge.
Since 2022, the EC-funded PathOS project (https://pathos-project.eu/) has worked to better understand and measure the academic, societal, and economic impacts of Open Science. As the project concludes, this presentation summarises its findings, tools, and recommendations.
Key topics include:
Current evidence of impact: Insights from three PRISMA-guided scoping reviews, covering over 700 studies. Findings from six PathOS case studies will also be discussed to build a picture of the known Key Impact Pathways for Open Science.
Indicators for impact: The PathOS Open Science Indicator Handbook (https://handbook.pathos-project.eu/) provides a “cook book” for measuring Open Science uptake and impact and discusses challenges in attributing impact. We present an overview of the handbook and plans for its sustainability.
Economic methodologies: PathOS is developing and testing a framework for cost-benefit analysis of Open Science interventions (initial concept: https://zenodo.org/records/10277642). Results from two large validation case studies will be available at the conference.
Challenges: Lessons learned include gaps in evidence generation, dataset availability, causal attribution, and synthesising broad Open Science impact insights. Discussion will includes how current developments like the Open Science Monitoring Initiative can contribute to ameliorating these difficulties.
Recommendations: PathOS is currently drafting recommendations for strengthening Open Science impact assessment in the future, which will be presented for the first time in this talk.
Few studies reach their full informative potential due to research and publishing inefficiencies, causing research waste. We collected practices that reduce waste at different stages of research and strategies that funders, publishers and institutions can apply to increase the use of best practices.
Paper Abstract
Scientific research is often conducted in a suboptimal manner, while many research projects also remain unpublished. This reduces the informative value of such research for the wider scientific and non-scientific audience and has been termed as ‘research waste’. Open scientific practices such as pre-registration, open data and similar have a strong potential to increase the efficiency and usability of research. Therefore, the aim of this study was to identify open, but also other research practices that decrease research waste as well as strategies, applied by funders, publishers, and research institutions, that encourage the application of such best practices. To this end, we conducted a hackathon in which 28 participants were given published literature on the topic of increasing research quality and decreasing research waste. They were asked to identify research practices that increase the efficiency and usability of research by improving one or more phases (relevance of research, study planning, reporting, publication, implementation of results). Participants also recorded strategies that promote such best scientific practices, as well as stakeholders in the scientific system (institutions, funders, publishers) who are responsible for implementing these strategies. Through the hackathon, we identified 13 relevant research practices and a large number of strategies to mitigate research waste. These strategies include development of infrastructure, support, training, incentivizing, policy making and policy compliance monitoring. This work provides a valuable foundation for further development of tools and guidelines to increase the efficiency of scientific research. The poster will be interactive to collect any additional relevant practices and strategies.
Quality assessment of primary studies is an essential part of evidence synthesis influencing the interpretation of evidence. This task is complicated by the exponential growth of research. We present a new checklist for identifying problematic studies and early exclusion from evidence synthesis.
Paper Abstract
The credibility of findings from evidence syntheses is becoming an increasing concern, as the unexpected amounts of retrieved studies, questionable data and inconsistent findings are slowly overwhelming the publication system and potentially biasing meta-analytic conclusions. We present a newly developed integrity assessment checklist and a pilot study to provide an overview of research integrity in special education research and assess the feasibility and generalizability of the checklist on other social sciences. The checklist consists of items aimed at evaluating the research integrity of the study and items that assess the business practices of the publisher. More specifically, we evaluate ethics statements, transparency in reporting, originality of the text, plausibility of the findings, consistency of statistical reporting within the text, and the editorial practices of the journal and the business model of its publisher. An expert panel provided feedback and suggestions for the items, after which the checklist was piloted by an end-user panel. Piloting was conducted on studies from a scoping review of educational interventions and later tested on several adjacent social science areas of research to evaluate generalizability. We expect that the checklist will enable researchers to conduct quick checks of studies to exclude problematic studies before they are included in thorough quality appraisal, which in turn will facilitate evidence synthesis and ensure the trustworthiness of included studies.
We present a collaborative, multi-stakeholder approach to support funders’ efforts to drive transparency in trials. We describe funders’ needs for supporting transparency, the potential of automated tools for monitoring, and co-creation workshops to inform novel strategies for improved transparency.
Paper Abstract
Transparency is core to responsible research: it allows the quality of research to be appraised, mitigates biases, and facilitates comprehensive evidence synthesis. While transparency is important in all areas of science, clinical trials warrant particular scrutiny because they involve human volunteers and inform medical decision-making. Yet, trials often lack transparency; for example, trial results may be delayed, unavailable, or poorly linked. Insufficient transparency undermines evidence synthesis and distorts our understanding of the medical evidence base.
Funders are uniquely positioned to drive improvements in research transparency by setting policies, monitoring funded projects, and supporting compliance. However, funders often lack a comprehensive overview of grantees’ adherence to transparency practices, or efficient workflows to monitor compliance. Alliances with metascientists as well as other types of institutions (e.g., registries, research organizations, regulators) offer the opportunity to better understand funders’ challenges and needs, drive technological and methodological advances, and build long-term infrastructures to shape scalable change towards transparency in clinical trials and beyond.
In this talk, we present a collaborative approach to support funders’ efforts to drive trial transparency. We describe the findings of an interview study on funders’ perceived role, existing measures, and needs around monitoring and supporting trial transparency. We showcase opportunities for funders to integrate automated tools into their monitoring workflows. Finally, we describe ongoing workshops, which bring together funders and other stakeholders to co-create novel solutions to improve trial transparency. Our work illustrates how metascience can build alliances within the community to promote innovation toward responsible research.
A UK Committee on Research Integrity project is generating evidence to inform recommendations for governance of research misconduct in UK Higher Education. Findings comprise information about scale, scope and perceived effectiveness of 24 international models for management of research misconduct.
Paper Abstract
Although there is no evidence base that intentional research misconduct is increasing in the UK, appropriate prevention and management of research misconduct is likely to support trust and confidence in research and the systems that produce it.
Definitions of research misconduct vary internationally, with some broad congruence because of global frameworks. However, systems and structures for management of research misconduct differ considerably, including varied legal arrangements and research environments.
In the UK, research misconduct is defined in the Concordat to Support Research Integrity as including fabrication, falsification, plagiarism, failure to meet legal, ethical and professional obligations, and various forms of misrepresentation. Effective management of research misconduct includes prevention of misconduct and robust processes to address instances when misconduct is alleged to have occurred.
As part of the UK Committee on Research Integrity’s work to develop a rigorous evidence base to underpin recommendations to the UK research sector, we commissioned RAND Europe to gather and analyse information about national and international governance when concerns of research misconduct in Higher Education are raised. One phase of work has been the scoping and characterisation of international approaches to the management of research misconduct allegations. In desk-based work, 24 country approaches have been described, including material relating to their effectiveness. Further scrutiny of approaches in 10 countries, selected for their diversity, includes interviews with stakeholders to understand policies, procedures and resourcing arrangements. This poster presents findings from the international work as a first step in development of recommendations for UK Higher Education.
Using a suite of qualitative methods UNDISCIPLINED explored how transdisciplinary research (TDR) is described and evaluated across six TDR funding programmes. We present a framework based on three facets of meaning:Partnerships, Values, and Impact, to help funders arrive at their optimal definition.
Paper Abstract
Transdisciplinary research (TDR) is about researchers making a difference to people's lives by working side by side with them. There is no universal definition of TDR and the term ‘transdisciplinary’ varies in meaning across contexts. TDR takes diverse forms across disciplines and fields of practice, driven by various motivations. Against this backdrop research funding organisations need to clearly convey what is meant by TDR in order to attract, identify and select appropriately designed projects. In the UNDISCIPLINED project we examined how TDR is described in funding programmes, and how peer reviewers are guided by funders to ensure that applications meet TDR evaluation criteria. Using iterative, collaborative methods, researchers and funders worked together to produce a literature review, document analysis and organisational case studies, centred on six funding agencies across numerous TDR programmes. Our findings demonstrate that TDR definitions consistently encompass three facets of meaning: Partnerships, Values, and Impact. These facets were weighted differently in each funding programme we examined. All programmes emphasised the need for a broad mix of perspectives on evaluation panels, which may include a combination of disciplinary perspectives, academic/non-academic stakeholders, or both, depending on the programme context. Evaluation criteria were closely aligned to TDR descriptions with funders’ supporting actions additionally building and binding these elements together with TDR programmes. We recommend that funders define what transdisciplinary research means for themselves and their communities, including the relative importance of Partnerships, Values, and intended Impacts, use mixed panels, and strengthen communication between themselves, researchers and knowledge users.
IDIBAPS launched its first comprehensive Open Science policy to build capacities among researchers through new tools, training, and education. We analyze perceptions, challenges, and impacts on the research community, highlighting outcomes and increased engagement with open science practices.
Paper Abstract
The Institute of Biomedical Research August Pi i Sunyer (IDIBAPS) in Barcelona has embarked on a transformative journey to foster Open Science. In 2022, IDIBAPS launched its first comprehensive Open Science policy, expanding beyond the previous regulation on open access to publications. The new policy was implemented through two main actions: 1) infrastructure, introducing new tools and platforms, and 2) support mechanisms, including training and education, aiming to build capacities among the research community.
Here, we outline the development and implementation of the Open Science policy, providing a comprehensive overview of our journey, including an in-depth analysis of perceptions and challenges faced by our researchers, and the initial impact of our efforts on the research community at IDIBAPS. Insights were gathered through questionnaires and monitoring of diverse open science practices among researchers. We highlight common themes and concerns raised by researchers and address the shift in perception and engagement in open access and research data management practices.
We aim to illustrate this process highlighting the outcomes and increased engagement with open access and data management practices. We also address common challenges and solutions encountered, advocating for the broader adoption of open science within research institutions.
Moving forward, IDIBAPS aims to refine its policy based on ongoing feedback and continue fostering a culture of openness and collaboration. We are committed to sharing our experiences, processes, analysis and best practices with the broader research community to support the global advancement of open science.
Transparency and reproducibility are known issues in science, yet open science practices are not always followed. By assessing reporting quality of cross-sectional psychological studies from 2023, we examine if it predicts raw data sharing and compare AI and human assessment.
Paper Abstract
Introduction: Open science practices, such as reporting quality and data sharing, are necessary for ensuring proper transparency and reproducibility. However, authors are often hesitant to share their data. With AI advancing, its role in simplifying research processes, such as data extraction, remains open for exploration. This study will examine whether reporting quality done by human predicts authors’ sharing of data differently than AI assessment.
Methods: Cross-sectional studies from 2023 from four Q1 psychological journals (two requiring STROBE guidelines and two not requiring them), are considered eligible. Emails requesting raw data will be sent to corresponding authors. The included articles’ reporting quality will be assessed by measuring the adherence to the STROBE guidelines. Two authors and ChatGPT 4o (OpenAI) will assess the reporting quality and human and AI assessment will be compared by calculating inter-rater reliability.
Results: Information concerning the adherence to STROBE guidelines from the 89 included articles will be extracted by the authors and the AI. A T-test will be used to assess whether there is a difference in reporting quality between those who have shared raw data and those who have not. Higher reporting quality of included articles is expected to be a predictor of sharing raw data. AI and human prediction is not expected to differ.
Conclusions: Findings could demonstrate the need for journals and other institutions to strengthen adherence to reporting guidelines and data sharing practices, and showcase the potential AI has in assisting with research processes to improve scientific practices.
This study examines whether ERC-funded researchers produce more novel contributions than unfunded but equally excellent applicants. Using novelty indicators, we find ERC funding does not enhance novelty, as funded researchers rely on established knowledge over unconventional ideas.
Paper Abstract
The European Research Council (ERC) funds frontier research with the goal of driving groundbreaking scientific advancements. Although ERC grants are designed to support high-risk, high-reward research, traditional bibliometric indicators—such as short-term citation counts or journal impact factors—often fail to capture novelty. As a result, how ERC funding promotes novel scientific contributions remains an open question. This study evaluates whether ERC-funded researchers produce more novel scientific contributions compared to similarly excellent, but unfunded, applicants. Using a Difference-in-Differences (DiD) approach with multiple time periods, we analyze ERC applicant data (2014–2020) from CORDA, matched with bibliometric data from OpenAlex and SciSciNet. We measure novelty through Foster’s Novelty metric, Atypical Combinations analysis, and Sleeping Beauty coefficients, capturing unconventional knowledge recombinations and delayed recognition effects. Our results indicate that ERC funding does not significantly increase the novelty of researchers' output compared to their unfunded counterparts. While ERC-funded scientists maintain high research productivity, their work tends to rely on established knowledge structures rather than pioneering unconventional combinations. Notably, the most novel outputs, as measured by Foster’s Novelty and Atypical Combinations, do not exhibit a systematic advantage for ERC-funded researchers. Additionally, Sleeping Beauty analyses reveal that breakthrough research often emerges independently of ERC support, suggesting that the grant’s selection process may favor safer, incremental advances over high-risk, disruptive ideas. These findings contribute to the ongoing debate on the role of competitive funding in fostering truly groundbreaking science and call for a reassessment of evaluation criteria to better support scientific novelty.
The Contributorship Collaboration is an international network aiming to improve researcher credit. Two projects facilitate using CRediT by providing translations in over a dozen languages and engaging with publishers. Our webapp, tenzing, helps manage and report author contributions.
Paper Abstract
The Contributorship Collaboration is an international and open network of researchers. The purpose is to improve how researchers are credited for their work. Two projects make it easier for researchers and publishers to use CRediT, the contributor roles taxonomy widely used by scientific journals to indicate what each co-author did. In our CRediT translation project, we have created quality translations of the CRediT roles and descriptions into over a dozen languages. At least one fluent speaker was involved in each translation. Another facet is outreach to publishers and journals associated with these language communities, not only to make them aware of the translations but also to inform them of contributorship more generally. Tenzing, a webapp we created is available at tenzing.club, helps collaborators on a research project manage author and contributor information and report it in manuscripts.
We apply network science and scientometrics to innovation policy. Multilayer-directed acyclic graphs and granular relational data are used to dynamically quantify knowledge flows, feedback loops, and bottlenecks, offering targets for policymakers and R&D managers’ intervention and evaluations.
Paper Abstract
Historically, analytical tools for understanding innovation systems have relied on abstractions, as the complexity and dynamics of R&D could never be fully modelled. With newly available tools, we combine large-scale relational scientometric datasets from Dimensions with network science to inform critical innovation policy decisions.
Building upon recent work, such as Ho et al.'s analysis of mRNA vaccine development published in Nature Biotechnology, we demonstrate how techniques like multilayer-directed acyclic graphs can attribute impact, identify rate-limiting steps, and reveal hidden dynamics in innovation networks. These methods are applied to real-world case studies, highlighting implications for policy issues such as strategic research portfolio management, infrastructure investment timing, and interagency coordination.
This paper bridges cutting-edge methodological developments with the practical needs of policymakers and R&D programme managers. We explore how innovation network analysis can guide more effective resource allocation across the R&D chain as technologies mature. We also identify new signals for tracking technology readiness and anticipating policy intervention points.
Ultimately, this research demonstrates a data-informed perspective on innovation policy. By showcasing novel analytical frameworks and grounding them in concrete use cases, we hope to stimulate further work at the intersection of network science, scientometrics, and innovation studies. The insights generated can inform the design of more effective, evidence-based strategies for nurturing and steering innovation ecosystems. This paper demonstrates the potential for data-driven approaches to innovation policy, enabled by integrating rich, linked datasets and advanced analytical techniques. We believe this approach will become the foundation for innovation policymakers in the future.
Research on Research (RoR) is growing fast but fragmented collaborations limit its impact. We created the ror-hub.org registry to offer a platform for sharing meta science projects and collaborations. Currently holding 240+ members and 61 projects, we aim for growth and impact through partnerships.
Paper Abstract
The rapid growth of the meta-science or Research on Research (RoR) discipline to enhance practices related to delivering, disseminating and implementing research, highlights the need for collaboration and evidence sharing to maximise research use and impact. However, isolated efforts have characterised RoR activity, contributing to fragmented evidence uptake and potentially research waste. To address this need, an online RoR registry and community platform (ror-hub.org) was launched in 2021.
Funded by the University of Southampton's Faculty of Medicine, with support from the NIHR and other research and funding organisations, the registry was created to serve the meta-science community. It currently has over 240 members and 61 registered studies and projects. Through its online community, online "Chatter" sessions, and a recent research festival, the hub has engaged with a network of over 900 people.
Our immediate goals for the RoR registry are of growth, building partnerships for long term sustainability and continued contribution to a positive research culture. By increasing the visibility of the registry and forging collaborations with a global network of funding organisations and Higher Education Institutions, we will address issues linked to the siloed practices that have characterised research on research activity. Joining efforts with other meta science initiatives, the registry will support the efforts of conducting transparent, collaborative, and impactful research that supports implementation of better research practices and the goals of the UK in consolidating its place in the global research landscape.
The Global Research Council's Responsible Research Assessment working group, has developed the "Dimensions of Responsible Research Assessment" framework. This set of 11 dimensions aims to guide funding agencies worldwide in implementing responsible research assessment practices.
Paper Abstract
In pursuit of a healthier global research culture that upholds the highest standards of rigor and integrity, the Global Research Council's Responsible Research Assessment (RRA) working group has introduced the "Dimensions of Responsible Research Assessment" framework. This comprehensive framework, developed through the collaboration of experts from countries across regions including Africa, Asia, Europe, North America, and South America, was launched at the GRC Annual Meeting in May 2024. It delineates 11 key dimensions that encapsulate funders' visions for responsible research assessment, providing a structured approach for funding agencies worldwide to integrate these principles into their evaluation processes. By embracing this framework, funding bodies can foster research environments that prioritize ethical practices, inclusivity, and excellence, thereby contributing to the advancement of knowledge on a global scale. The international composition of the working group underscores the applicability and relevance of the framework, reflecting a collective commitment to enhancing research assessment practices across different contexts. This initiative not only serves as a guide for funders but also as a catalyst for ongoing discussions and reforms in research assessment, aiming to align evaluation methods with the evolving dynamics of the global research landscape.
The interdisciplinary nature of the data science workforce extends beyond the notion of traditional data scientists. Successful teams require a range of technical expertise, domain knowledge and leadership skills. To reinforce a team-based approach, we recommend diversifying data science roles.
Paper Abstract
Data science is an interdisciplinary and collaborative field. The specific skills required for any data science initiative depend on the discipline, industry and the context within which data-informed solutions are developed. Effective data science teams should therefore be composed of diverse specialists who can flexibly combine their expertise to develop innovative solutions that address both societal and industry needs.
To build and strengthen the data science workforce, institutions, funders, and policymakers must invest in developing and diversifying these specialised professional roles, fostering a resilient data science ecosystem for the future. By recognising the diverse specialist roles that contribute to interdisciplinary teams, organisations can leverage deep expertise across a broad range of skills. This approach enhances responsible decision-making and fosters innovation at every level.
Building on existing standards and professional personas for AI and Data Scientists, and drawing upon examples of diverse professionals from The Alan Turing Institute (the UK's national institute for data science and AI), this paper/talk will share evidence-based recommendations to support the professionalization of different specialist roles. We further provide a detailed Skills and Competency Framework for Research Community Manager roles, offering a template for articulating skills and competencies for other newly emerging roles.
Our overarching aim is to shift the perception of data science professionals away from the traditional view of the lone "data scientist" and toward a competency-based model of multiple specialized roles. Through collaboration within a data science project team, members complement and combine their skills—each crucial to the success of data science initiatives.
We investigate how AlphaFold and other AI methods shape scientific discovery, focusing on research productivity, exploration of less-studied areas, and translational outcomes. Our large-scale analysis reveals shifts in structural biology and beyond, offering early insights into AI’s broader impacts.
Paper Abstract
Artificial intelligence has seen rapid diffusion across science. Proponents highlight the potential for more efficient and effective scientific discovery, while others raise concerns that AI leads to a narrowing of scientific research. AlphaFold, Google DeepMind’s AI-driven system for protein structure prediction, a long-standing problem in structural biology, has emerged as a high-profile example. In this paper, we examine AlphaFold’s impact in the broader context of AI’s potential to accelerate discovery, diversify research portfolios, and spur innovation. Leveraging a large-scale dataset of publications, patents, protein structures, and clinical trials, we employ difference-in-differences to compare AlphaFold’s effects to other frontier developments, both AI-based and non-AI-based, in structural biology.
Our findings indicate that AlphaFold is associated with notably higher experimental output, especially on novel protein structures. These gains are concentrated among experienced researchers and principal investigators, who also benefit from modest upticks in publication volume and citation metrics. While AlphaFold has not yet led to a notable increase in patenting activity compared to established non-AI methods, clinical citations linked to AlphaFold publications exceed those of many other AI-based approaches. This trend hints at its growing relevance for disease-related inquiries.
By mapping shifts in the organisms and protein structures targeted by researchers who use AlphaFold, we reveal how AI-enabled predictions can lower barriers to exploring less-studied domains. Beyond AlphaFold, our results speak to the broader capacity of AI methods to transform scientific practices, from productivity to translational applications. This underscores the need for continuous monitoring and policy support to harness AI’s promise in science.
What is the effect of concealing researchers' and institutions' identities in a first evaluation stage in order to focus on the proposal's content? What external factors influence the decision-making process, regardless of the proposal's quality?
Paper Abstract
Multiple studies have shown that the main sources of bias in the peer-review process are related to the applicant's personal characteristics and institutional affiliation. We present our experience managing a project peer-review assessment process that includes three evaluation stages: first, individual peer-review assessment with blinded CVs; second, individual revision of the assessment with unblinded CVs by the same reviewer; and third, assessment of the highest-rated proposals by an ad hoc committee. Our research questions are: What is the effect of concealing researchers' and institutions' identities in the first evaluation stage? What external factors influence peer-reviewers' decision-making process? We address these questions through a retrospective observational study analyzing 5,002 individual proposal evaluations. The primary variable of analysis was the change between the first (blinded) and second (unblinded) assessments. We analyzed factors associated with changes, taking into account the characteristics of proposals, reviewers, and researchers. A qualitative content analysis was conducted to assess the reasons for changes. The analysis revealed that in 19% of the evaluations, reviewers changed their second assessment, either upgrading (12%) or downgrading (7%) their initial rating. Our findings suggest that changes in the second assessment were highly correlated with positive evaluations of the principal investigator's or research team's experience. These changes also seemed to be influenced by the total grant amount requested, the reviewer's world region, and the principal investigator's gender. Peer-review procedures that initially focus solely on the content of a research proposal offer a promising way to reduce some common biases related to researchers' characteristics.
Impact policies ‘work’ – they promote impact as a value and goal amongst academic. But interviews with 90 AI-focused academics across three countries shows how impact policies also sharpen tensions within and between disciplinary specialisms, raising questions about policy effects and effectiveness.
Paper Abstract
Impact policies aim to shift academic incentives and practices away from merely production of knowledge towards realising potential socioeconomic benefits. We interviewed 90 academics from across the ‘hard’ and ‘soft’ sciences in Australia, UK and USA, all specialising on some aspect of artificial intelligence (AI), to investigate shifts in academic culture. One of the key findings relates to a split amongst the ‘hard’ scientists. For some, delivering/realising impact has become the core driving force of their work and their sense of academic identity and purpose, while others consider impact subservient to or secondary to more ‘traditional’ commitments to one’s immediate disciplinary community. This divide does not only affects impact engagement but has broader effects on academics’ research and educational practices. Another important but more nuanced finding relates to international differences in how impact policies affect academia. Impact policies interact with distinctive system features in ways which can lead to specific but unpredictable challenges arising within sub-groups. For example, impact policies appear to be sharpening divisions and debates amongst UK computer scientists about the field’s role in and relation to wider society. And for some, but especially for social scientists in the Australia context, there was a strong sense of disparity between impact rhetoric and reality, with impact activities perceived as not being genuinely understood, supported or valued by the sector. Overall, there is no uniform ‘impact of impact’ at either national or disciplinary level. Impact policies must be continuously analysed at relatively disaggregated levels to understand their effects and effectiveness.
This research on research work presents a portfolio-based methodology designed to guide the thematic focus of a biomedical research call. By comparing research efforts and capacity with disease burden, the approach identifies funding gaps and supports evidence-based strategic decision-making.
Paper Abstract
Global challenges in health demand targeted research; hence, funders are increasingly deploying mission-oriented funding strategies. Assessing whether these instruments align with health and societal needs might not be straightforward, and a portfolio-level approach can help, rather than cherry-picking project evaluations. In this way, targeted and strategic funding can help shape the demand for biomedical research oriented to societal and health needs.
This work presents a methodology designed to shape the thematic focus of a biomedical research call, ensuring that research investments align with societal needs. By comparing research capacity with disease burden, this approach identifies funding gaps and provides a flexible, data-driven framework for strategic decision-making.
Using publicly available data and straightforward indicators, the methodology offers evidence-based insights to support prioritization in science policy. It highlights strengths and misalignments in research investment, enabling funding agencies and policymakers to optimize resource allocation. The methodology’s adaptability allows it to be tailored to different strategic priorities, while its visual component enhances accessibility, fostering discussion among diverse stakeholders.
Here we present an application of this approach in prioritizing research funding across five major disease groups, comparing research capacity and disease burden in Catalonia. The methodology proves particularly valuable when combined with participatory processes, SWOT analyses, refined definitions of research priorities, or prospective scenario analyses.
In conclusion, this methodology provides evidence-based insights to support policymakers in designing funding mechanisms that optimize resource allocation and maximize societal impact. This work contributes to advancing methodologies for evaluating the effectiveness of research funding strategies in addressing societal needs.
This study explores how targeted funding for healthcare professionals influences research agendas and evidence-based practice. Using a nursing research program as a case study, we assess its impact on research capacity, professional legitimacy, healthcare innovation, and nursing policy development.
Paper Abstract
Nursing research has traditionally played a limited role in how biomedicine addresses complex healthcare challenges. Targeted funding schemes for nursing research offer critical benefits: advancing both disciplinary and practice-oriented knowledge, strengthening research capacity and legitimacy, and supporting institutions, conditions, and methods often underfunded in conventional biomedical programs.
This study evaluates a Catalan research program (2017–2022) that funded 127 nurse-led projects to integrate research into clinical practice. Using a mixed-methods approach, combining quantitative analyses of administrative data, qualitative insights from 30 in-depth impact case studies, and engagement with policy makers and healthcare professionals, we analysed the program’s bidirectional impact on nursing research, healthcare practices and policy strategies. Key findings include:
• Enhancing knowledge mobilization. Funded projects produced not only peer-reviewed publications but also diagnostic tools, care protocols, and educational materials leading to the implementation of new interventions, programs, and organizational changes.
• Strengthening professional roles. It supported the consolidation of the dual professional profile of nurse-researchers, increasing legitimacy within healthcare institutions.
• Informing institutional change and policy. The evaluation influenced the Catalan nursing research strategy, led to the development of a regional nursing research map, and integrated research objectives into professional development policies.
This case study demonstrates how evaluating funding models informs institutional transformation, strengthens transdisciplinary research cultures and provides scalable frameworks for embedding research into health systems, aligning funding with clinical needs, and involving practitioners in knowledge production. This model demonstrates how funders can design research programs that shorten the impact pathway, strengthen professional research cultures and shape evidence-based policy.
To ensure the next generation of researchers upholds scientific standards, training is essential. FORRT is developing a pedagogically-informed, evidence-based, self-guided program to support educators. This session will present the program, gather feedback, and explore ways to expand its reach.
Paper Abstract
Ensuring that future researchers and consumers of science are well-versed in Open Science is essential for maintaining and advancing scientific standards. Educators and mentors play a crucial role in this process by providing a solid foundation in Open Science principles. To support them, the Framework for Open and Reproducible Research Training (FORRT) is developing a pedagogically-informed, evidence-based, self-guided training program, called POST-Ed (a Positive, Inclusive and Participatory Program for Educators).
This initiative aims to equip educators with structured, accessible resources for teaching Open Science effectively. The program consists of three core modules, designed with a positive, participatory, and inclusive approach. By fostering engagement and accessibility, FORRT ensures that educators from diverse academic backgrounds can integrate Open Science practices into their teaching.
A key aspect of this initiative is collaboration with the Open Science community. By involving researchers and educators throughout the development process, we aim to refine and enhance the program to best serve the needs of those teaching Open Science.
In this session, we will present an overview of the program, gather insights on areas for improvement, and explore strategies to broaden its impact. Our goal is to ensure that Open Science training reaches educators across various academic fields, empowering them to cultivate a new generation of researchers committed to transparency and reproducibility.
Using a data-led approach, one society based publisher will share their research integrity data, recorded between 2019 and 2024, to showcase the scale and pace of change that has taken place. They will then reflect on the policy changes that have been made as a result.
Paper Abstract
This presentation will present an analysis of high-level data from recent research integrity cases at a medium-sized society publisher, highlighting key trends and the subsequent policy changes implemented to address these issues.
We analyse data recorded between 2019 and 2024 (inclusive) relating to research integrity allegations, both pre and post publication. This data shows what types of problems are emerging and at what scale.
Our findings indicate a notable increase in cases involving author changes, the use of recommended reviewers, and the need for careful vetting of reviewer reports. Additionally, the use of generative artificial intelligence by both authors and reviewers has introduced new complexities, necessitating updates to existing policies.
In response, we have developed pragmatic approaches to ensure the integrity of the peer review process. By sharing our experiences and strategies, we aim to foster a collaborative dialogue on best practices for maintaining research integrity in an increasingly complex and technologically advanced environment.
This could work as a poster or a talk/presentation.
Shared data allows research to be reproduced and new insights to be generated, but participants opting out of having their data shared can significantly limit this potential. As such, this study explores techniques to synthetically recreate non-shared data, ensuring better reproducibility and reuse.
Paper Abstract
Recent traction in open science and data sharing has led to many granting agencies requiring data to be publicly shared at the time results are published. This, in theory, leads to greater reproducibility of study results and allows for novel insights through secondary data analysis. However, the notion of publicly shared data may exacerbate existing concerns about participant privacy, and participants included in the original analyses may not consent to having their data shared publicly, a circumstance which we refer to as post-analytic attrition (PAA). The sharing of partial data sets due to PAA significantly limits potential for reproducibility and reuse; however, there are few other options for researchers who must respect the wishes of both their grant agencies and their participants. The present study introduces three techniques to supplement partially shared data sets, and examines the extent to which each approach improves reproducibility and potential for reuse. Publicly available data were repeatedly sampled to represent varying levels of PAA and rates of differential PAA by participant demographics. Covariance-based simulation, model-based simulation, and machine learning were used to recreate the non-shared data as a supplement to the original dataset. Using this synthetically generated data, we then attempted to reproduce the original study findings and recover its initial structure. Results were compared across the simulation techniques and levels of PAA/differential PAA. Future directions are also discussed, with particular focus on balancing participant privacy and data reuse, as well as prioritizing reproducibility versus general reuse.
The TIER2 project, funded under Horizon Europe, introduces Reproducibility Management Plans (RMPs) embedding reproducibility directly into research workflows from the planning stage.
Paper Abstract
Research reproducibility remains a critical challenge across disciplines, requiring innovative solutions that go beyond traditional compliance-driven data management approaches. The TIER2 project, funded under Horizon Europe, addresses this challenge through the introduction of Reproducibility Management Plans (RMPs): a novel enhancement to existing Data Management Plans (DMPs) that embeds reproducibility practices directly into research workflows from the planning stage.
This talk presents findings from the TIER2 RMP pilot, which evaluates the integration of reproducibility elements into the widely used DMP template released by Science Europe. Through co-creation activities with researchers, the pilot identified key gaps in current DMPs, including the need for detailed data provenance, methodological transparency, and improved domain-specific guidance. Participants reported that DMPs often serve administrative purposes rather than being practical instruments for enhancing research integrity. Based on these insights, the pilot proposes key improvements to transform DMPs into valuable research assets, including structural enhancements, metadata and documentation clarity, domain-specific guidance and standards. TIER2 is now developing a dedicated RMP template, to be implemented via the ARGOS platform (argos.openaire.eu). This template will capture essential metadata for machine-actionable RMPs, ensuring that reproducibility is systematically embedded in research planning.
Verifying dataset accessibility in repositories like Figshare, Zenodo, and OSF is manual and inefficient. To address this, we introduce SciCiteCheck, an automated tool leveraging APIs such as re3data, which will be released under the AGPL 3.0 license.
Paper Abstract
In academic manuscripts, authors sometimes cite datasets stored in repositories such as Figshare, Dataverse, Zenodo, and the Open Science Framework (OSF); however, no efficient method exists to verify that these repositories contain the cited data beyond manual inspection.[https://oakland.libguides.com/c.php?g=1404215&p=10392907]. To address this issue, we have developed SciCiteCheck, a tool that checks dataset citations for accessibility status.
With SciCiteCheck, researchers can automatically query cited datasets and determine their accessibility status—whether they are open, under controlled access, or subject to embargo. This tool makes use of publicly available APIs from major data repositories like re3data, which is the most comprehensive source of reference for research data infrastructures [https://medium.com/towards-data-science/data-repositories-for-almost-every-type-of-data-science-project-7aa2f98128b].
SciCiteCheck offers a functional back-end and a friendly front-end interface to facilitate easier verification of dataset citations. Preliminary results suggest that SciCiteCheck helps researchers efficiently verify dataset citations, check accessibility status, and ensure proper referencing of datasets in their work. The tool will be released under the AGPL 3.0 license, ensuring open access and transparency.
We present the Researcher Profile, a platform developed within the GraspOS project to display researchers' contributions, emphasizing Open Science practices. It integrates data from OpenAIRE Graph and ORCID and incorporates emerging indicator frameworks.
Paper Abstract
The reform of research assessment is a central priority in European policy, emphasising transparency, inclusivity and responsible evaluation. . The GraspOS project (graspos.eu)—Next Generation Research Assessment to Promote Open Science—advances this effort by developing tools and services to support institutions in implementing responsible research assessment practices.
At the core of this work is the Researcher Profile, a platform designed to showcase researchers’ diverse contributions, while prioritising Open Science principles. It integrates the OpenAIRE Graph, an open research database for scholarly communication that connects research outputs, projects, and collaborations across institutions. Additionally, it aggregates preprints from major repositories, ensuring comprehensive visibility of early-stage research. Additionally, it includes data on 110K organizations with ROR identifiers, reinforcing institutional recognition and interoperability.
The platform builds on emerging indicator frameworks from recent initiatives and projects that aim to reward diverse research skills, career trajectories, and contributions to research impact. It aligns with the principles of CoARA and DORA, allowing for qualitative assessment through Narrative CVs, Open Science indicators and activities, and the illustration of impact activities.
The Researcher Profile integrates quantitative assessments/measures with qualitative narratives, supporting a fairer and more inclusive evaluation of research careers. It highlights Open Science contributions and ensures that research impact is recognised beyond publications and citations.
This work highlights our collaboration with ROR, ORCID, and OSF.io, emphasising integrated open infrastructure for responsible assessment. Our approach strengthens transparency, context-aware evaluation frameworks, promotes diverse career paths, and aligns with European efforts for a fair and inclusive research ecosystem.
Bridging the gap between academia and industry is key to maximising the societal impact of research. However, there is limited evidence on how to do this effectively. This handbook presents experimental ideas to address barriers related to motivation, capabilities, resources and matching.
Paper Abstract
Bridging the gap between academia and industry is essential to maximise the societal impact of the knowledge generated in research institutions. However, policymakers across the OECD frequently highlight the insufficient levels of science commercialisation and business-university collaboration. There is a prevailing belief that significant value remains untapped, and that more can be done to ensure scientific discoveries lead to greater societal impact. While numerous activities aim to address this gap, evidence of their effectiveness remains limited.
Experimental research methods, such as randomised controlled trials (RCTs), can help organisations to generate better evidence and develop more effective programmes. Yet it remains uncommon to see knowledge valorisation activities tested in a rigorous way – the scientific method is seldom applied to science commercialisation activities.
This handbook offers a compilation of experimental ideas to address common barriers in the science commercialisation journey. Specifically, it presents experimental ideas tackling motivation, capabilities, resources, and the matching process from both the researcher and business perspectives. These ideas aim to inspire and guide policymakers, programme implementers, and researchers in developing and evaluating their strategies to increase university-industry collaboration and foster a more dynamic innovation ecosystem.
For the full handbook see: https://www.innovationgrowthlab.org/resources/experimenting-with-university-industry-collaboration
Awesome Systematic Reviews is a collaborative GitHub repository curating resources on evidence synthesis, focusing on automation tools. It compiles tool features, documentation, and use cases, supporting researchers by centralizing key materials and encouraging community contributions.
Paper Abstract
Systematic reviews are fundamental to evidence-based decision-making, particularly in healthcare, public policy, and other research-driven fields. However, their rigorous methodology makes them highly time-consuming and resource-intensive. Recent advancements in Artificial Intelligence (AI) and automation are transforming this process, accelerating tasks such as literature screening, data extraction, and synthesis while maintaining methodological rigor. Awesome Systematic Reviews is an open, community-driven GitHub repository (https://github.com/pawljmlo/awesome-systematic-reviews) designed to centralize and curate resources related to evidence synthesis, with a focus on automation tools, including AI-powered solutions.
This initiative systematically catalogs tools that support various stages of systematic reviews and guideline development, including searching, study screening, data extraction, and risk-of-bias assessment. To ensure researchers have access to up-to-date information, we are conducting a scoping review of existing reviews that describe these tools. The repository compiles essential details such as tool functionalities, links to documentation, relevant research papers, pricing models (open-source vs. paid), and the specific phases of evidence synthesis in which each tool can be applied. Beyond automation tools, the repository also serves as a hub for guidelines, events, and research networks related to evidence synthesis.
A defining feature of Awesome Systematic Reviews is its collaborative, open-access nature. Anyone can contribute by submitting issues or pull requests to suggest new tools, provide feedback, or share experiences. As AI and automation continue to evolve, our goal is to maintain a dynamic, community-driven resource that grows alongside the field. We invite researchers, developers, and methodologists to engage with Awesome Systematic Reviews, helping to build a shared knowledge hub.
Evidence syntheses generate vast amounts of structured data through extraction, quality assessment, and GRADE evaluations — yet these data remain largely inaccessible. REX-QA aims to collect, standardize, and share these datasets to enhance AI training, reproducibility, and decision-making.
Paper Abstract
Systematic reviews and guideline development require extensive data extraction as well as quality and certainty of evidence assessment (e.g., GRADE). However, these structured datasets are rarely shared beyond the original research teams, limiting their utility for metascience, artificial intelligence (AI) training, and automation in evidence synthesis. The lack of transparency also makes it difficult to evaluate consistency in assessments across different teams.
REX-QA (Review EXtraction and Quality Assessment) data corpus is an initiative to collect, standardize, and openly share these valuable datasets. By gathering data from completed systematic reviews and guideline projects, REX-QA (https://osf.io/9pbrs/) promotes transparency, reproducibility, and machine learning applications in evidence synthesis. We also propose a standardized data extraction template to facilitate structured data storage in accessible formats (e.g., CSV).
To launch this effort, we are sharing data from our own previous studies, demonstrating the feasibility and benefits of structured data sharing. We welcome contributions of extracted data both before and after team consensus, allowing researchers to analyze variations in evidence assessments and grading practices. This can help identify inconsistencies, improve standardization, and refine AI-assisted evidence appraisal.
In the era of large language models (LLMs) and AI-driven systematic reviews, structured, high-quality datasets are essential for training reliable models. REX-QA provides a foundation for improving AI-powered evidence synthesis while promoting open science. We invite researchers, systematic reviewers, and AI developers to contribute data, refine standards, and explore innovative applications, fostering a more transparent and efficient future in evidence synthesis.
Students are often overlooked in reform discussions. Drawing on five case studies, this talk shows how student participation enhances scientific rigor, drives innovation, and enriches research culture. We present key strategies for effectively integrating students into reform initiatives.
Paper Abstract
Students are often overlooked in discussions about improving the scientific system, yet they are essential stakeholders in shaping the future of research. As potential scientists of tomorrow, students play a crucial role in academic ecosystems—both as learners and as active participants in the research process. However, traditional perspectives on scientific innovation and the reform of the academic system tend to focus on researchers, funders, and publishers, leaving students underrepresented in conversations.
In this talk, we present five case studies of student engagement in psychology and more generally social science research, showing how student involvement improves scientific quality, fosters innovation, and strengthens research culture. These cases deal with the involvement of students at university level, within the framework of cross-university research projects as well as at the national level. They illustrate how structured collaborations between students and researchers contribute to methodological rigor, transparency, and inclusivity. Based on these examples in our own field of expertise, we draw success factors that can be utilised across different scientific fields to promote reform efforts with the help of students. We also provide actionable insights for institutions wishing to integrate students more effectively into the research enterprise.
Some retracted work continues to be cited post-retraction. Some papers cite this work without awareness of its retracted status, but others do (e.g. to attempt to replicate). I present a tool that detects citation intentionality with high accuracy (95%) to help understand if retractions are working.
Paper Abstract
The retraction of unreliable work is an essential part of self-correcting science. However, many papers continue to receive citations after retraction, some even amassing more citations after being retracted than before. While some of these citations come from scholars that are unknowingly relying on discredited research, not all post-retraction citations are a cause for concern. For example, papers may cite a retracted work when they are attempting a replication or pointing to an example of poor or fraudulent science to avoid. Distinguishing between these very different categories of post-retraction citations is an essential step towards understanding how effective retractions are as a corrective measure. Currently there is no way to detect at scale whether papers that cite retracted work are doing so intentionally. Prior work attempting this classification used a rule-based heuristic method that I demonstrate results in many intentional citations being misclassified (high false negative rate). I present an alternative model trained on ~1000 works from the PubMed Open Access Corpus that achieves a 95% classification accuracy rate (F1=0.95) during 10-fold cross validation. Using the more accurate classifier and manual annotation I also present an improved benchmark dataset and discuss how this model can be used to estimate the prevalence of unintentional post-retraction citations in the scientific literature broadly.
*7-1-25: This original abstract reported a 95% accuracy and accuracy score. After cleaning duplicated data that was originally missed, the new pilot results are (10FoldCV to train on 80% of data, test accuracy on held-out 20%) 91.7% and F1 0.91
This study employed NLP and ML methods to examine research evidence use in legislatures. Transformer-based models, NER, and citation mapping were utilised to detect research uptake and trace its sources, while an evidence classification framework was developed to enable sentence-level analysis.
Paper Abstract
A working assumption in legislative science advice (LSA) literature is that transparency and robust, research evidence-based scrutiny are critical to effective policymaking. Yet, there are no comprehensive models examining research evidence use in legislatures. This study leveraged NLP and ML methods to systematically detect, trace, and analyse explicit and tacit research evidence use in legislatures at scale.
The study focused on British parliamentary scrutiny since 2022. Over 20,000 records were processed, including debate transcripts, written questions, committee documents, and in-house research (IHR) briefings with a corresponding dataset of cited academic and policy sources. We applied transformer-based models and NER to detect instances where parliamentarians used IHR briefings verbatim. Citation mapping then traced matched sentences to their likely original sources. Finally, a classification framework enabled analysis of how parliamentarians contextualise different evidence.
A qualitative examination of parliamentary speech identified seven types of evidence. Every detected instance from an IHR briefing was assigned one of seven labels. To explore contextualisation, the preceding and following sentences were also extracted and classified. These three-sentence sequences were recorded in a new dataset to identify patterns.
Our findings reveal patterns of selective citation, a strong reliance on high-impact publications, and the strategic framing of research. Additionally, metadata analysis identified instances where IHR briefings were relevant but not used.
This research advances metascience by providing a scalable, AI-driven framework to assess research impact in governance and enhance transparency. It informs strategies to increase research visibility and uptake, offering actionable insights for policymakers, researchers, and knowledge brokers.
Machine learning (ML) diagnosis of medical images attracts a lot of attention, yet progress in clinical practice has not been proportional. Despite larger datasets and open-source tools, reproducibility is a problem. In this talk we dive into these problems, and what changes are needed in future.
Paper Abstract
Machine learning (ML) diagnosis of medical images has attracted a lot of attention recently, with some claims of surpassing expert-level performance, yet progress in clinical practice has not been proportional.
The increased popularity is often explained by two developments. First, there are several large publicly available datasets. Second, open source ML toolboxes allow development of algorithms with less domain expertise.
Despite these seemingly ideal conditions for reproducibility, there are several issues, in this talk I will highlight two. One issue is that large sample sizes are not a panacea. There is a tendency to expect that a clinical task can be “solved” if the dataset is large enough. However, not all clinical tasks translate neatly into ML tasks, and creating larger datasets can come at the expense of data quality.
Another reason is the availability of data and code, plus the option to “infinitely” run experiments (with different subsets of data, different parameters, etc), which creates an illusion of reproducibility. Even if a code repository is available, it might not be clear (1) what data was used - as data might not be cited, or can be a derivative of a public dataset, or (2) or how many different experiments were also run, but that are not in the repository.
In this talk we dive deeper into these problems and hopefully, with the help of the audience, also explore some solutions. We will also touch upon various incentives in ML and academia that interact with these findings.
In discussing learning, observing, applying and overcoming questionable research practices in our own research, we build a narrative raising awareness about experiencing and internalizing perverse incentive structures stemming from scientific culture and institutions.
Paper Abstract
Why would we engage in practices that undermine the reliability of science? If the purpose of science is to produce valid, systematic and reliable knowledge, QRPs are basically anti-science. Yet, they remain prevalent. We believe that both (a) the desire to produce findings that can be recognized, published and cited in the scientific community, but also (b) through transmission, mimicry, sanction and reification drives these behaviors. By starting at the beginning to understand why we became scientific researchers in the first place we will demonstrate that our intentions were to produce scientific knowledge - knowledge that could be used to address real world (mis)understandings and problems. At some point we came under the delusion that QRPs would produce this knowledge. In this panel we use autoethnographic analysis of our experiences as researchers to understand QRPs and build a narrative to shed light on the inner-workings of perverse incentive structures from the researcher perspective. We will focus our panel toward discussions of the publish-or-perish paradigm and how we experience this through our cultural and institutional interactions in science. We experience this in the job search process for example, where publications are at the center of our potential for getting an interview. In the funding search process as well, publications are the key credentials to measure our worth as scientists. In sharing and discussing our stories, we hope to address the question asked in the call for proposals of “how can metascience help us to understand and improve institutionalised research cultures”.
Software is critical to research and innovation: it is crucial to gain understanding of such ‘hidden’ building blocks that underpin our digital infrastructure. UKRI, the Software Sustainability Institute, OpenUK and Invest in Open will discuss their work exploring this fundamental infrastructure.
Paper Abstract
A 2016 report ‘Roads and Bridges’: the unseen labor behind our Digital Infrastructure’ (Nadia Eghbal) states that ‘Free, publicly available source code is the infrastructure on which all of digital society relies’. With the rise of AI, the recognition of the impact of data-led initiatives, and increasing risks around cybersecurity, we must understand the ‘hidden’ building blocks that underpin our digital infrastructure. The software landscape is diverse and complex, due to a range of funding and licensing models, and often maintained only due to the goodwill of developers. Should we rethink how we support this important infrastructure?
ADORE.software: the Amsterdam Declaration on Funding Research Software Sustainability states “The crucial role of software in research is becoming increasingly apparent, as is the urgent need to sustain it and to invest in the people who develop and maintain it. Research software sustainability is vitally important for the reproducibility of research outcomes and has a direct bearing on the process of research, including the efficient use of financial and human resources.”
In 2024 UK Research and Innovation (UKRI) commissioned projects to better understand the nature of the software that underpins research and innovation. The evaluation included assessment of how software is funded, licensing regimes, areas of risk, opportunities to support open ecosystems, and a first step to map the software that underpins UKRI’s investments. Organisations such as Invest in Open and OpenUK are also working to research and mobilise funding and strategies to bolster the underlying systems that open research depends on.
Researchers often face arbitrary choices when analyzing data, limiting the generalizability of a single outcome. Multiverse analysis mitigates this by exploring multiple pathways but still risks bias. We propose a principled approach using crowdsourcing to enhance transparency and objectivity.
Paper Abstract
When processing and analyzing empirical data, researchers regularly face choices that may appear arbitrary (e.g., how to define and handle outliers). If one chooses to exclusively focus on a particular option and conduct a single analysis, its outcome might be of limited utility. That is, one remains agnostic regarding the generalizability of the results, because plausible alternative paths remain unexplored. A multiverse analysis offers a solution to this issue by exploring the various choices pertaining to data processing and/or model building, and examining their impact on the conclusion of a study. However, even though multiverse analyses are arguably less susceptible to biases compared to the typical single-pathway approach, it is still possible to selectively add or omit pathways. To address this issue, we outline a novel, more principled approach to conducting multiverse analyses through crowdsourcing. We also provide a worked-out illustration featuring the Semantic Priming Across Many Languages project, thereby demonstrating its feasibility and its ability to increase objectivity and transparency.
Short Abstract
This page displays all the content accepted for the poster boards throughout the conference and the session on the Tuesday evening.
Long Abstract
Click on any poster below to read more detail as to the content.
Accepted posters
Session 1 Tuesday 1 July, 2025, -