Short-Term Pain for Long-Term Gain: Adaptive Experiment with Post-Commitment Reward Shift

2026-07-25

AI Summary

This work addresses the misalignment between short-term optimal decisions and long-term rewards in adaptive experimentation, particularly when reward shifts may occur during the commitment phase. The authors propose the RAEC algorithm, which reserves resources during the exploration phase to jointly minimize short-term regret and accurately identify the long-term optimal arm, and extend it to settings with structural priors and combinatorial commitments. Theoretically, they provide the first tight minimax characterization of the trade-off cost between short-term performance and long-term commitment, revealing that under structural priors, identifying changes in reward rankings is more critical than estimating the magnitude of shifts. They further introduce the ROSCOC algorithm, which directly maps exploration history to a committed combinatorial action. The proposed algorithms achieve tight regret upper bounds across various parameter regimes, and numerical experiments demonstrate their superiority over baseline methods.

Abstract

Decision-makers in learning environments face a dilemma when their short-term optimal actions may not favor their long-term benefits the most. To understand the fundamental tradeoff behind the dilemma, we study adaptive experimentation with post-commitment reward shifts. During an experiment phase, the decision-maker may adaptively test multiple options; during a subsequent commitment phase, the decision-maker must commit to a single option, whose reward may differ from its pre-commitment reward. We propose the Reserved Arm Eliminations for Commitment (RAEC) algorithm, which reserves a predetermined portion of the experiment phase to identify the best post-shift option while using the remaining rounds to minimize short-run regret. We establish regret upper bounds for RAEC across all parameter regimes and matching minimax lower bounds, providing a tight characterization of the cost of balancing short-term performance and long-term commitment. We also study two extensions. With prior structural knowledge linking pre- and post-shift rewards, we show that correctly identifying the ranking-changing component of the shift is more important than estimating its absolute magnitude. For settings with concave commitment rewards and portfolio choice, we develop the Reserved Online Stochastic Convex Optimization for Commitment (ROSCOC) algorithm, which directly converts its reserved exploration history into a commitment portfolio and achieves tight regret bound. Finally, we also conduct numerical experiments which confirm that our proposed algorithms achieve the desired regret predicted by our theory, and also outperform other baseline algorithms.

Problem

• adaptive experimentation
• post-commitment reward shift
• short-term vs long-term tradeoff
• commitment phase
• regret minimization

Innovation

• adaptive experimentation
• post-commitment reward shift
• regret minimization
• arm elimination
• online convex optimization