🤖 AI Summary
This study addresses the limitations of conventional variance reduction methods in switchback experiments on online platforms, where clustered structures and temporal autocorrelation often invalidate standard assumptions. The authors develop a hierarchical simulation framework to systematically evaluate the performance of CUPED, CUPAC (a machine learning–based covariate adjustment approach), doubly robust estimators, and cluster-robust standard errors across diverse experimental conditions—including varying numbers of clusters, levels of autocorrelation, and spillover effects. Through sensitivity analyses accounting for cross-cluster interference, they quantify each method’s behavior in terms of false positive rates, confidence interval coverage, standard error reduction, statistical power, and minimum detectable effect sizes. The work culminates in a practical decision map that delineates the applicability boundaries and trade-offs of these variance reduction techniques, highlighting fundamental constraints imposed by temporal and cluster dependence.
📝 Abstract
Switchback experiments and other clustered randomized designs are widely used on online platforms, but the clustered, time-dependent nature of these designs can make standard variance reduction methods behave differently than in standard A/B tests. We evaluate design-aware variance reduction methods for switchbacks -- CUPED, CUPAC (ML-based covariate adjustment), and doubly robust (DR) estimators -- relative to a baseline switchback analysis with cluster-robust standard errors. Through a hierarchical simulation framework that varies key regime parameters -- number of clusters, cluster-size imbalance, within-cluster autocorrelation, carryover, and predictive signal strength -- we evaluate validity (false positive rate and confidence interval coverage) and efficiency (standard error reduction, power, and minimum detectable effect as a function of run length). We also include a sensitivity analysis for cross-cluster spillovers to quantify bias and inference degradation under mild interference. The primary outcome is a practitioner-oriented regime map: when CUPED, CUPAC, or DR are most beneficial, and when time and cluster dependence and finite-cluster effects limit improvements.