Beyond Basic A/B testing: Improving Statistical Efficiency for Business Growth

2025-05-13

AI Summary

Industrial A/B testing often suffers from low statistical power due to small sample sizes, non-Gaussian response distributions, and ROI-driven decision-making—challenging conventional t-tests. To address these issues, we propose a doubly robust generalized U-statistic framework that unifies ROI sensitivity modeling, distributional robustness, and small-sample inference for the first time. We rigorously establish its asymptotic normality and semi-parametric efficiency bound. The framework integrates regression adjustment, generalized estimating equations, Mann–Whitney U statistics, and zero-truncated U-statistics, balancing theoretical rigor with engineering scalability. Extensive simulations and empirical evaluations across multiple business scenarios demonstrate a 20–40% improvement in statistical power over state-of-the-art baselines, significantly accelerating data-driven product iteration and growth decisions.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

The standard A/B testing approaches are mostly based on t-test in large scale industry applications. These standard approaches however suffers from low statistical power in business settings, due to nature of small sample-size or non-Gaussian distribution or return-on-investment (ROI) consideration. In this paper, we propose several approaches to addresses these challenges: (i) regression adjustment, generalized estimating equation, Man-Whitney U and Zero-Trimmed U that addresses each of these issues separately, and (ii) a novel doubly robust generalized U that handles ROI consideration, distribution robustness and small samples in one framework. We provide theoretical results on asymptotic normality and efficiency bounds, together with insights on the efficiency gain from theoretical analysis. We further conduct comprehensive simulation studies and apply the methods to multiple real A/B tests.

Problem

• Improving statistical power in A/B testing with small samples
• Addressing non-Gaussian distribution challenges in standard t-tests
• Enhancing ROI consideration in robust statistical frameworks

Innovation

• Regression adjustment for non-Gaussian distributions
• Doubly robust generalized U for ROI
• Man-Whitney U for small sample sizes