🤖 AI Summary
This paper addresses the selection problem of identifying the optimal $m$ units out of $n$ to maximize the true total value, under highly noisy and heteroscedastic observations. We propose an empirical Bayes estimation-and-selection framework. Our key theoretical contribution is the first rigorous proof that, when the prior estimation error is $O_p(r_n)$, the selection regret is bounded by $O_p(r_n^2)$—a rate shown to be tight under the given parametric assumptions. The method integrates parametric prior modeling, asymptotic regret analysis, and calibration using real-world online experimentation data. Extensive evaluation across over 4,000 online A/B tests demonstrates that our approach achieves high-precision identification of optimal interventions with minimal experimental overhead, substantially outperforming naive thresholding methods.
📝 Abstract
We study the problem of selecting the best $m$ units from a set of $n$ as $m / n o alpha in (0, 1)$, where noisy, heteroskedastic measurements of the units' true values are available and the decision-maker wishes to maximize the aggregate true value of the units selected. Given a parametric prior distribution, the empirical Bayes decision rule incurs $O_p(n^{-1})$ regret relative to the Bayesian oracle that knows the true prior. More generally, if the error in the estimated prior is of order $O_p(r_n)$, regret is $O_p(r_n^2)$. In this sense emph{selection} of the best units is fundamentally easier than emph{estimation} of their values. We show this regret bound is sharp in the parametric case, by giving an example in which it is attained. Using priors calibrated from a dataset of over four thousand internet experiments, we confirm that empirical Bayes methods perform well in detecting the best treatments with only a modest number of experiments.