🤖 AI Summary
This paper studies the infinite-armed nonstationary multi-armed bandit (MAB) problem, where arm mean rewards drift over time in an unknown, arbitrary manner and no prior knowledge of nonstationarity parameters is available. To address this challenge, we propose a fully parameter-free online learning framework: first, a statistically significant reward drift detection mechanism enables adaptive identification of critical drifts; second, a black-box transformation coupled with randomized elimination adapts classical finite-arm algorithms to the infinite-arm setting; third, under mild reservoir distribution assumptions, we establish, for the first time, a minimax-optimal regret bound and uncover the distinctive phenomenon that “increasing rewards do not exacerbate learning difficulty.” Theoretical analysis yields a tight, adaptive regret rate depending solely on rotting-type nonstationarity—significantly improving upon existing parameter-dependent approaches.
📝 Abstract
We study an infinite-armed bandit problem where actions' mean rewards are initially sampled from a reservoir distribution. Most prior works in this setting focused on stationary rewards (Berry et al., 1997; Wang et al., 2008; Bonald and Proutiere, 2013; Carpentier and Valko, 2015) with the more challenging adversarial/non-stationary variant only recently studied in the context of rotting/decreasing rewards (Kim et al., 2022; 2024). Furthermore, optimal regret upper bounds were only achieved using parameter knowledge of non-stationarity and only known for certain regimes of regularity of the reservoir. This work shows the first parameter-free optimal regret bounds for all regimes while also relaxing distributional assumptions on the reservoir. We first introduce a blackbox scheme to convert a finite-armed MAB algorithm designed for near-stationary environments into a parameter-free algorithm for the infinite-armed non-stationary problem with optimal regret guarantees. We next study a natural notion of significant shift for this problem inspired by recent developments in finite-armed MAB (Suk&Kpotufe, 2022). We show that tighter regret bounds in terms of significant shifts can be adaptively attained by employing a randomized variant of elimination within our blackbox scheme. Our enhanced rates only depend on the rotting non-stationarity and thus exhibit an interesting phenomenon for this problem where rising rewards do not factor into the difficulty of non-stationarity.