🤖 AI Summary
This paper addresses sequential experimental design under high sampling costs: how to adaptively allocate units to two treatments and dynamically determine stopping time to maximize expected welfare net of sampling costs (i.e., minimize worst-case regret). Methodologically, it proposes a unified stopping rule based on the product of the estimated average treatment effect and sample size exceeding a cost-dependent threshold, coupled with Neyman allocation for unit assignment. Theoretically, it establishes—first rigorously—that Neyman allocation is minimax-optimal for sampling, and characterizes the optimal stopping time via this threshold-crossing condition. The policy remains minimax-optimal under both parametric and nonparametric settings and in the small-cost asymptotic regime. It unifies Wald’s sequential hypothesis testing and best-arm identification as special cases. The solution admits a closed-form analytic expression and retains robust optimality even in the zero-cost limit.
📝 Abstract
We study sequential experiments where sampling is costly and a decision-maker aims to determine the best treatment for full scale implementation by (1) adaptively allocating units between two possible treatments, and (2) stopping the experiment when the expected welfare (inclusive of sampling costs) from implementing the chosen treatment is maximized. Working under a continuous time limit, we characterize the optimal policies under the minimax regret criterion. We show that the same policies also remain optimal under both parametric and non-parametric outcome distributions in an asymptotic regime where sampling costs approach zero. The minimax optimal sampling rule is just the Neyman allocation: it is independent of sampling costs and does not adapt to observed outcomes. The decision-maker halts sampling when the product of the average treatment difference and the number of observations surpasses a specific threshold. The results derived also apply to the so-called best-arm identification problem, where the number of observations is exogenously specified.
