🤖 AI Summary
This work addresses the inefficiency of traditional sequential experimental design in the predict-then-optimize paradigm, which stems from its neglect of downstream decision loss. The authors propose a decision-oriented sequential design method that directly optimizes the performance of downstream linear optimization problems by introducing a novel measure of directional uncertainty that does not require an optimization oracle. Unlike conventional approaches that prioritize predictive accuracy alone, this method targets decision quality while maintaining computational efficiency and enjoying strong consistency and convergence guarantees. Theoretical analysis demonstrates that the approach achieves earlier stopping times under broad distributional assumptions. Empirical evaluation on a real-world task allocation problem involving large language models shows significant improvements over existing methods.
📝 Abstract
We consider the sequential experimental design problem in the predict-then-optimize paradigm. In this paradigm, the outputs of the prediction model are used as coefficient vectors in a downstream linear optimization problem. Traditional sequential experimental design aims to control the input variables (features) so that the improvement in prediction accuracy from each experimental outcome (label) is maximized. However, in the predict-then-optimize setting, performance is ultimately evaluated based on the decision loss induced by the downstream optimization, rather than by prediction error. This mismatch between prediction accuracy and decision loss renders traditional decision-blind designs inefficient. To address this issue, we propose a directional-based metric to quantify predictive uncertainty. This metric does not require solving an optimization oracle and is therefore computationally tractable. We show that the resulting sequential design criterion enjoys strong consistency and convergence guarantees. Under a broad class of distributions, we demonstrate that our directional uncertainty-based design attains an earlier stopping time than decision-blind designs. This advantage is further supported by real-world experiments on an LLM job allocation problem.