🤖 AI Summary
This study addresses the challenge of efficiently guiding treatment allocation in a main experiment using a small-scale pilot, avoiding efficiency losses from noise or excessive conservatism. The authors propose the Conditional Minimax Regret (CMR) rule, which optimizes assignment probabilities in a two-stage design by leveraging confidence sets constructed from limited pilot data, thereby balancing robustness and adaptivity. The CMR rule preserves, with high probability, the worst-case guarantees of balanced designs while asymptotically converging to the Neyman allocation as the pilot sample size grows, achieving the minimax regret rate. The approach naturally extends to multi-arm and stratified settings. Simulations demonstrate that CMR substantially outperforms feasible Neyman allocation when the pilot is small—avoiding its severe precision loss—while recovering most of its efficiency gains in large samples.
📝 Abstract
Experimenters often run pilots, but how much a small pilot should shape the main-wave design has no settled answer. This paper shows how noisy pilot evidence should guide treatment assignment probabilities in two-wave experiments. Two canonical rules mark the extremes. Balanced assignment guards against worst cases but ignores evidence that one arm is noisier. Feasible Neyman allocation adapts, but with a finite pilot it can overreact to noise, producing arbitrarily large precision losses. We propose a Conditional Minimax Regret (CMR) rule that minimizes worst-case regret over a finite-sample confidence set for the treatment and control variances. CMR retains balance's worst-case protection with high probability, converges to the Neyman allocation as the pilot grows, and attains the minimax-regret rate up to constants. It extends to multi-arm and stratified designs, and simulations calibrated to four field experiments show it avoids feasible Neyman's severe small-pilot losses while capturing most of its large-pilot gains.