🤖 AI Summary
This study addresses the exploration–exploitation trade-off in sample size determination for test-and-roll experiments over a finite population, proposing a welfare-oriented approach that requires no prior information. The method introduces a worst-case marginal benefit (WMB) criterion, which overcomes the shortcomings of traditional minimax regret by comparing the worst-case marginal gain from adding a matched pair of observations against its marginal exploration cost. Leveraging Gaussian approximations for Bernoulli outcomes and building on the known-variance Gaussian model, theoretical analysis shows that—excluding pathological cases—the optimal experiment size is approximately one-third of the total population. This “one-third rule” offers practitioners a simple, robust, and prior-free guideline for experimental design.
📝 Abstract
This paper studies sample-size design for finite-population test-and-roll experiments, where a decision-maker first conducts an experiment on $m$ units and then assigns the remaining $N-m$ units to the treatment that performs better in the experiment. We consider welfare-aware sample-size choice, which involves an exploration-exploitation tradeoff: larger experiments improve the rollout decision but impose welfare losses on experimental units assigned to the inferior treatment. We show that the standard absolute minimax regret criterion can lead to implausibly small experiments by over-penalizing exploration in its worst-case objective. To address this limitation, we propose the Worst-case Marginal Benefit (WMB) rule, which compares the worst-case marginal benefit of adding one more matched pair to the experiment with the corresponding marginal exploration cost. We establish a simple rule-of-thirds benchmark. For Bernoulli outcomes, after excluding pathological cases, the WMB criterion yields the optimal sample size of $m \approx N/3$ through a Gaussian approximation. For Gaussian outcomes with a known common variance, the same benchmark arises exactly. These results provide a prior-free and practically implementable guide for welfare-based sample-size design.