Short Abstract

This panel provides a platform for cross-disciplinary researchers to exchange ideas, experiences, and knowledge about AI for research statistics and evaluation to fill the gap between AI and its responsible use in metascience. The panel session includes opening remarks, power talks, and open Q&As.

Long Abstract

The tremendous success of large language models (LLMs) has been revolutionising the paradigm of scientific research. Leveraging AI’s capabilities in optimally perceiving data, learning patterns, and summarising historical knowledge with insightful conclusions, to achieve evidence-based decision support has become a common pursuance of the society. We have observed exciting LLM use cases in broad metascience practices, while the gap between AI’s underlying logic and its responsible use still exists. This panel aims to provide a platform for cross-disciplinary researchers to exchange ideas, experiences, and knowledge about AI for research statistics and evaluation.

The 90-min panel session will be organised as follows:

• Welcome and Opening Remarks (10 minutes)

---- Yi Zhang: AI for Scientometrics

• Panel Discussion: AI for Research Statistics and Evaluation (70 minutes)

---- Moderator: Yi Zhang

---- Power Talks (each panellist, 10 minutes)

---- ------ Gunnar Sivertsen: LLMs for OECD Research Aims

---- ------ Lin Zhang: LLMs for Gender Difference in Research Aims

---- ------ Mike Thelwall (University of Sheffield, UK): LLMs for Peer Review Evaluation

---- ------ Robin Haunschild (Max Planck Institute for Solid State Research, Germany): LLMs for Post Peer-review Recommendations

---- Open Questions (30 minutes)

---- ------ Which part of AI could bring the core benefit to our community?

---- ------ What are the key capabilities of utilising AI for research evaluation?

---- ------ Any ethical concerns?

• Closing and Wrap-up (10 minutes)

---- Moderator: Mengjia Wu

---- One takeaway per panellist

---- Closing Remarks: Lin Zhang