Always-On Experimentation

2026-09-29

AI Summary

This study addresses the challenge of false discovery rate (FDR) control in dynamic experiments accelerated by generative AI, where treatment groups are continuously added and removed. To this end, this work proposes a time-uniform error control method that supports arbitrary stopping times and predictable scheduling. By constructing supermartingale test statistics within a betting-based testing framework, the proposed approach enables rigorous statistical decision-making in dynamic environments. The core contribution lies in establishing that the proposed test not only achieves almost-sure optimal growth rates theoretically but also introduces a novel hypothesis testing paradigm that combines flexibility with statistical rigor for continuously running online experiments.

Abstract

Generative AI has dramatically accelerated the rate at which new treatments---from novel pharmaceuticals to online marketing campaigns---can be conceived and deployed. As a result, modern experimentation platforms often run continuously, with treatments added as they are ready and removed when they underperform. We formalize this "Always-On" experimental setting, in which treatments can be dynamically generated, added to, and removed from a running experiment, and study the statistical problem of deciding whether to accept or reject each treatment while controlling for the false discovery rate. We develop sequential tests that achieve time-uniform Type-I error control under arbitrary stopping times and "predictable" treatment schedules. Our approach builds on the testing-by-betting framework: we construct test supermartingales for testing the average treatment effect of each treatment, and show that the construction of these test supermartingales is growth-rate optimal in an almost-sure sense.

Problem

• Always-On Experimentation
• False Discovery Rate
• Sequential Testing
• Generative AI
• Type-I Error Control

Innovation

• Always-On Experimentation
• Sequential Testing
• False Discovery Rate
• Testing-by-Betting
• Test Supermartingales