Ranking by Lifts: A Cost-Benefit Approach to Large-Scale A/B Tests

2024-07-01

arXiv.org

AI Summary

This paper addresses the fundamental problem of maximizing expected profit under a cost-weighted false discovery rate (FDR) constraint in large-scale A/B testing. We develop a decision-theoretic statistical inference framework that innovatively incorporates the local false discovery rate (lfdr) into a cost-benefit ratio ranking criterion, yielding a greedy empirical Bayes procedure equivalent to a knapsack optimization problem with an FDR constraint. The method is theoretically optimal under asymptotic conditions and empirically validated for stability and effectiveness in finite samples. Experiments on real-world data from the Optimizely platform demonstrate that our approach significantly increases business profit compared to conventional FDR control methods, while rigorously maintaining statistical reliability. Thus, it achieves a principled unification of statistical rigor and operational objectives.

Technology Category

• Search and Optimization: Learning to Search
• Machine Learning: Online Learning & Bandits
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Security and Privacy: Large-scale security measurements
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

A/B testers that conduct large-scale tests often prioritize lifts as the main outcome metric and want to be able to control costs resulting from false rejections of the null. This work develops a decision-theoretic framework for maximizing profits subject to false discovery rate (FDR) control. We build an empirical Bayes solution for the problem via a greedy knapsack approach. We derive an oracle rule based on ranking the ratio of expected lifts and the cost of wrong rejections using the local false discovery rate (lfdr) statistic. Our oracle decision rule is valid and optimal for large-scale tests. Further, we establish asymptotic validity for the data-driven procedure and demonstrate finite-sample validity in experimental studies. We also demonstrate the merit of the proposed method over other FDR control methods. Finally, we discuss an application to data collected by experiments on the Optimizely platform.

Problem

• Maximizing expected profit under cost-weighted false discovery rate constraint
• Ranking A/B experiments by expected lift to cost ratio
• Controlling false discoveries while optimizing business value

Innovation

• Empirical Bayes with greedy knapsack algorithm
• Ranking by expected lift-to-cost ratio
• Controlling cost-weighted false discovery rate