On the Benefit of Blocking for Online Experiments with Skewed Data

2026-10-07

AI Summary

This study addresses the estimation bias and noise amplification induced by highly skewed data in online A/B testing. We propose a unified framework integrating stratified randomization, fixed-weight blocking, and within-block winsorization. Specifically, the fixed-weight blocking mechanism precisely isolates extreme values, effectively circumventing efficiency-weighting bias while fully preserving tail causal effects, and within-block winsorization further suppresses local anomalous fluctuations. Although the precision gains are modest, this approach substantially corrects the estimation bias of the average treatment effect and effectively controls variance inflation. Our findings reveal the structural advantages of blocking strategies in handling skewed distributions, offering a robust experimental design solution for high-noise settings.

Abstract

This paper advocates for using blocked (stratified) assignment in online A/B tests to handle highly skewed population data. While blocking yields only modest precision gains (5-10%) due to the limits of discretization, the authors demonstrate it offers two crucial structural benefits over post-hoc statistical adjustments. First, fixed-weight blocking correctly targets the true average treatment effect, avoiding the severe bias introduced by efficiency-weighted alternatives. Second, blocking localizes extreme outliers, enabling targeted within-block winsorization that effectively controls noise without destroying the tail-end treatment effect.

Problem

• online A/B testing
• skewed data
• average treatment effect
• extreme outliers
• blocked assignment

Innovation

• blocked assignment
• skewed data
• average treatment effect
• winsorization
• online A/B tests