🤖 AI Summary
This study addresses the limitations of current replication research, which relies on binary judgments to estimate replicability rates yet struggles to reliably distinguish true replicability due to inexact experimental replications and the absence of a shared data-generating mechanism. The authors propose two formal modeling frameworks—shared latent variables (as a benchmark) and conditional independence (as an operationalization)—to characterize statistical heterogeneity in inexact replications and its impact on replicability estimation. Through Bayesian hierarchical modeling, identifiability analysis, and quantification of heterogeneity, coupled with a reanalysis of the Many Labs 4 dataset, they demonstrate that standard methods fail to account for between-study heterogeneity, resulting in an irreducible lower bound on the variance of replicability estimates and a systematic underestimation of uncertainty. Consequently, aggregated replicability rates across heterogeneous studies lack stable interpretation, and conventional replication approaches are insufficient to reliably substantiate claims of a “replication crisis.”
📝 Abstract
Replication studies estimate the replicability rate of scientific results by aggregating binary verdicts of experiments. Exact replications are rarely attainable, so most replication sequences are non-exact. Experiments differ in ways that matter and do not share a single data-generating process. We formalize two statistical interpretations of non-exactness. In a shared latent rate (benchmark) model, experiments are exchangeable and depend on a common random replicability rate. In a conditionally independent rates (operational) model, each experiment has its own replicability rate drawn from a population distribution. Under the benchmark model, even small variability among replicability rates induces an irreducible variance floor on the estimated mean replicability rate that no amount of replication can eliminate. Under the operational model, the degree of non-exactness is not identifiable from standard replication data, because one binary verdict per experiment carries no information about between-experiment heterogeneity. Researchers cannot tell which precision regime they are in or whether high- and low-replicability sequences can be distinguished in principle. The usual data structure cannot support reliable demarcation between "replicable" and "not replicable" results and systematically understates uncertainty, making high- and low-replicability sequences appear discriminable when they are not. We show how common sources of heterogeneity amplify these problems and demonstrate practical consequences in a reanalysis of Many Labs 4. Aggregating replicability rates across heterogeneous literatures produces averages that conflate incommensurable regimes and lack a stable interpretation. Replicability rate is not a reliable demarcation criterion. The replication crisis, if there is one, cannot be established by the methods used to declare it.