Short abstract

This paper analyzes publication patterns in economics by looking at the number of hypotheses with results reported in studies from the AEA RCT Registry. By tracking hypothesis-level outcomes in over 300 registered trials, it provides unprecedented insight into the size of the file drawer problem.

Long abstract

We developed an approach to standardize and record the hypotheses of studies registered in the AEA RCT Registry. We apply this approach to the registration of 300 studies registered in the AEA Registry between 2015 and 2017. This corresponds to thousands of hypotheses, and we proceed to search for the corresponding results in the published literature, publicly available working papers, and other research reports. This is among the largest follow-up studies in economics aimed at measuring publication bias and selective reporting, and is novel in focusing on results at the research hypothesis level. Preliminary analyses show substantial heterogeneity in the number of registered hypothesis, many of which we were unable to find in the published literature up to nine years later. This result suggests that preregistration alone may not solve the problem of selective reporting. In addition to measuring publication bias, we also develop an innovative approach to standardized reporting of research results in economics – a practice already widely used in other fields, including medicine and public health – that could be of interest to journals, funders, and professional associations.

Short abstract

Mass resignations from academic journals can yield new Breakaway journals, founded by resigning editorial boards. Zombie journals are restaffed with new personnel, retaining their histories and brands. This talk analyzes three recent case studies of competing Breakaway/Zombie journal pairs.

Long abstract

Mass editorial board resignations from journals are an increasingly widespread phenomenon in academic publishing. Breakaway journals are new journals founded by resigning editorial boards to compete with the journal they abandoned. Publishers often respond to these mutinies by re-staffing journal editorships and editorial boards to preserve the journal brand. Zombie journals are established journals that persist following mass resignations. Breakaway and Zombie journals provide an opportunity to empirically examine how removing part of a journal’s community from its brand affects subsequent publishing behaviors and niches. This article analyzes early case studies of editorial board mutinies in linguistics, bibliometrics, and mathematics. Difference-in-difference analyses of pre- and post-mutiny journals reveal that the Breakaway journals quickly established market share at the expense of their Zombie counterparts. Post-mutiny Zombie journals exhibited declines in publishing volume, citations received and authorial status. In both linguistics and bibliometrics, Breakaway journals were largely led and patronized by North American and European authors. This created vacancy chains at the Zombie journals, filled by an influx of China-affiliated authors, revealing unique institutional and cultural scientific reward structures in Chinese science vis-à-vis traditionally leading Western counterparts. Results suggest that academic capital associated with journal brands is mutable and appropriable, providing opportunities for institutional entrepreneurs to reshape academic publishing, as well as their broader intellectual fields. Varying competitive strategies of Breakaway and Zombie journals are discussed. Academic journals are Janus-faced in that they entail often-powerful institutions and brands, but are also comprised of the substantive labor and skills of people.

Short abstract

Why do researchers peer review? Analyzing administrative data from physics journals, we find that reviewers learn by reviewing: They are over twice as likely to cite reviewed papers.

Long abstract

Why do researchers contribute their time to peer review despite increasing pressures on their schedules? While previous research has emphasized altruistic motives and reputational benefits, we propose that personal learning represents a significant yet overlooked incentive. We investigate whether and when researchers acquire knowledge by peer-reviewing using administrative data from the Institute of Physics comprising 104,306 reviewer-manuscript pairs from 60 peer-reviewed journals.

Our analysis, involving within-manuscript comparisons between reviewers and those who were invited to review but were unavailable, reveals that reviewing a manuscript more than doubles the likelihood of citing it within three years (1.3% → 3.1%, p<0.001). Notably, this effect is moderated by both geographic and intellectual proximity. While invited reviewers from the same country as the manuscript's corresponding author who do not review cite the paper at high rates (3.0%), invited reviewers from a different country who do review cite the paper similarly, suggesting that reviewing compensates for geographical barriers. Reviewing has its largest effect on citing when the paper is intellectually very close to the reviewer’s recent work, suggesting that peer reviewing does little to break intellectual barriers.

Overall, reviewers learn by reviewing, helping explain their continued participation in the process. The results also illuminate how knowledge diffuses across geographic and intellectual boundaries.

Short abstract

What forms of transparency, over what data, might make academic publishing more robust? We build on researchers’ use of publishers’ data to explore how transparency might enhance rigour, challenge commercial pressures and improve the governance of academic publishing.

Long abstract

This panel builds on the experience of researching and writing a paper recently published in QSS called 'The Strain on Scientific Publishing'. This paper analysed trends in publishers activity based in part on all indexed academic publications from 2016-2022 and web-scraped information on the vast majority of all published papers by the leading eight publishers. But the process of collecting our data raises important questions about the meaning and nature of transparency. What information is required for publishers, and researchers, to be held to account for their actions? And what sorts of institutions might best provide oversight of this behaviour? Or, to put this differently, publishing wrestles with the dilemmas of publishing rigorously produced work, as well as the need to be profitable. These conflicting interests need to be effectively governed. In what contexts, and in what ways, might improved transparency provide that oversight? This panel explores the methods which can render publishing more transparent, the ethical dilemmas these raise, and the oversight of the data resulting that might contribute to producing an academic publishing ecosystem its contributors would enjoy building.

Short abstract

Whose papers end up in the file-drawer? Tracking 126K papers rejected by 62 physics journals reveals that authors from Western countries are more likely to ultimately publish these papers, and to publish faster and with less revising. A potential mechanism is access to procedural knowledge.

Long abstract

Scientific progress relies on making research contributions public, typically through journal publication, enabling others to build on them. However, publishing often requires overcoming one or more rejections. This study examines how scientists' differential responses to manuscript rejection shape both published knowledge and the “file drawer” of unpublished research. Analyzing 126K manuscripts rejected by 62 STEM journals published by the Institute of Physics Publishing, we document several new empirical facts. Controlling for manuscript quality (proxied by peer review recommendations) and comparing authors from Western and non-Western countries, we find that authors based in Western countries are 5.9% more likely to publish rejected manuscripts elsewhere, publish 25 days faster, revise 6% less, and change co-authors 11.6% less. A plausible contributing mechanism is geographic differences in access to procedural knowledge – how to interpret feedback from these journals, revise, and resubmit elsewhere. Although exploratory surveys of rejected authors are inconclusive, we find that post-rejection outcomes are better for corresponding authors with prior publishing experience and Western co-authors, both of which may proxy procedural knowledge. These findings imply that the “file drawer” contains a disproportionate number of ideas from non-Western countries, partly due to disparities in procedural knowledge.