Paper Short Abstract

We assessed reliability and measurement reporting in 77 Many Labs replications and related original articles. We evaluated the impact on replication informativeness and success, and advise minimum measurement reporting standards and informed replication choice.

Paper Abstract

In the wake of the “replication crisis”, many scientific reform initiatives have focused on promoting replication. However, if the measurement in the original study lacks reliability and construct validity, the findings of a replication study will not be informative. Recent studies have found issues with the measurement in both original and replication studies. However, the impact these issues have on the informativeness of replications remains unclear. We assessed the reliability and measurement reporting practices of 77 measures within 56 Many Labs replications and related original articles (Ebersole et al., 2016, 2020; Klein et al., 2014, 2018), as well as their impact on the replication success. First, we observed that reliability was not sufficient in each context for several measures. Second, measures were rarely accompanied with reliability and validity evidence. Third, questionable measurement reporting practices - but not reliability - were associated with lower replicability in our sample. These results corroborate existing findings that construct validity in published research is often unknown, which may reduce the informativeness of replication research. We discuss how academic stakeholders can improve of measurement through consistent use of minimum standards for construct validity, and by including reported measurement information as part of the decision process in deciding what to replicate.