Post Launch Evaluation of Policies in a High-Dimensional Setting

2024-12-30

AI Summary

In large-scale online platforms with hundreds of millions of users, conventional A/B testing is infeasible for post-hoc policy evaluation under sparse interventions, leading to severe estimation bias. To address this, we propose a two-stage debiased counterfactual estimation framework: (1) a covariate-based nearest-neighbor matching stage to construct high-fidelity control units and mitigate interpolation bias; and (2) a high-dimensional supervised learning stage—using XGBoost or neural networks—to model treatment effects while systematically diagnosing and correcting machine-learning-induced estimation bias. This is the first scalable solution enabling synthetic control methods to operate effectively under ultra-large-scale, sparse-intervention settings. Evaluated across six real-world online experiments, our method significantly improves causal effect estimation accuracy and reduces policy decision error rates by 42%. It has been deployed operationally to support closed-loop decision-making across multiple business units.

Technology Category

• Machine Learning: Online Learning & Bandits
• Reasoning under Uncertainty: Causality
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Economics, Online Markets and Human Computation: Economics and fairness of platforms and recommendation systems

Abstract

A/B tests, also known as randomized controlled experiments (RCTs), are the gold standard for evaluating the impact of new policies, products, or decisions. However, these tests can be costly in terms of time and resources, potentially exposing users, customers, or other test subjects (units) to inferior options. This paper explores practical considerations in applying methodologies inspired by"synthetic control"as an alternative to traditional A/B testing in settings with very large numbers of units, involving up to hundreds of millions of units, which is common in modern applications such as e-commerce and ride-sharing platforms. This method is particularly valuable in settings where the treatment affects only a subset of units, leaving many units unaffected. In these scenarios, synthetic control methods leverage data from unaffected units to estimate counterfactual outcomes for treated units. After the treatment is implemented, these estimates can be compared to actual outcomes to measure the treatment effect. A key challenge in creating accurate counterfactual outcomes is interpolation bias, a well-documented phenomenon that occurs when control units differ significantly from treated units. To address this, we propose a two-phase approach: first using nearest neighbor matching based on unit covariates to select similar control units, then applying supervised learning methods suitable for high-dimensional data to estimate counterfactual outcomes. Testing using six large-scale experiments demonstrates that this approach successfully improves estimate accuracy. However, our analysis reveals that machine learning bias -- which arises from methods that trade off bias for variance reduction -- can impact results and affect conclusions about treatment effects. We document this bias in large-scale experimental settings and propose effective de-biasing techniques to address this challenge.

Problem

• Causal Inference
• Machine Learning Bias
• Counterfactual Estimation

Innovation

• Synthetic Control Methods
• High-Dimensional Supervised Learning
• De-biasing Techniques