🤖 AI Summary
This paper studies the delay-aware contextual bandit problem: a learner dynamically selects a subset of arms under stochastic contexts, where each selection incurs a random delay and a context-dependent arm request cost; the total time horizon is determined by the cumulative delays of selected subsets. The problem is formulated as a semi-Markov decision process (SMDP), the first to jointly model context dependence, arm-specific request costs, and stochastic delays. We propose an online algorithm derived from the Bellman optimality equation and establish a tight $O(sqrt{T})$ regret bound under a realizability assumption—matching the optimal rate of classical contextual bandits. The theoretical analysis is rigorous, and extensive experiments on synthetic benchmarks and a movie recommendation dataset validate both the algorithm’s empirical effectiveness and its theoretical guarantees.
📝 Abstract
We introduce a novel extension of the contextual bandit problem, where new sets of arms can be requested with stochastic time delays and associated costs. In this setting, the learner can select multiple arms from a decision set, with each selection taking one unit of time. The problem is framed as a special case of semi-Markov decision processes (SMDPs). The arm contexts, request times, and costs are assumed to follow an unknown distribution. We consider the regret of an online learning algorithm with respect to the optimal policy that achieves the maximum average reward. By leveraging the Bellman optimality equation, we design algorithms that can effectively select arms and determine the appropriate time to request new arms, thereby minimizing their regret. Under the realizability assumption, we analyze the proposed algorithms and demonstrate that their regret upper bounds align with established results in the contextual bandit literature. We validate the algorithms through experiments on simulated data and a movie recommendation dataset, showing that their performance is consistent with theoretical analyses.