Short abstract

Heterogeneity of meta-analytical effects is often overlooked. We review three sets of psychological meta-analyses (total 1,207 meta-analyses). 22%-41% of them do not report heterogeneity at all. When reported, heterogeneity is high but rarely considered in the authors’ conclusions.

Long abstract

In psychology, the perception that meta-analyses represent conclusive evidence is widespread. Yet, recent findings contradicting some of the most prominent meta-analyses of the discipline indicate that meta-analytical evidence may be largely overstated, distorting research results and leading practitioners astray. Part of the reason might be that meta-analysts often base their conclusions on statistical significance (p-values) and the size of the average effect, overlooking other crucial information, such as the heterogeneity of effects. Here, we review three datasets: a pool of 100 most cited meta-analyses across five subfields of psychology (applied, clinical, developmental, educational, social), a pool of 714 meta-analyses published in social-psychology journals, and a pool of 393 meta-analyses from the PSYNDEX/PsychOpen CAMA database. Preliminary results indicate that 21.50% of the most cited meta-analyses and 40.99% of meta-analyses from social psychology journals do not report any information about heterogeneity at all. When heterogeneity is reported, it tends to be high, with average I^2 values of 69.00%, 70.20%, and 80.93% in the three datasets, and the average tau = .10, .20, and .21, respectively, accompanying an average effect of r = .27, .21, and .20, respectively. This elevated heterogeneity, combined with a small average effect, complicates the interpretations of results, which is rarely considered in the conclusions formulated by the authors of the meta-analyses. These omissions might contribute to low reliability of meta-analytical results in psychology.