π€ AI Summary
This work investigates the fundamental learning speed limits of mean-based algorithms under bandit feedback with an unknown time horizon, and explores their relationship with no-regret algorithms. We propose two novel algorithms: one generalizing the Ξ΅-greedy strategy and another extending the mean-based Exp3 algorithm to settings with unknown horizon. Our theoretical contributions include establishing, for the first time, a lower bound on the Ξ³β sequence in mean-based algorithms, thereby proving the existence of an inherent limit on learning speed, and revealing a non-trivial intersection between mean-based and no-regret algorithms. Empirical results demonstrate that although our proposed algorithms converge slightly slower, they achieve competitive performance, confirming the feasibility of simultaneously satisfying both mean-based estimation and no-regret properties.
π Abstract
Mean-based algorithms are a class of online learning algorithms that assign low probability to actions with low average rewards. Recent work indicates these algorithms converge favorably to serially undominated actions, which approximate Nash equilibria in economic games. However, empirical studies also show slower convergence compared to established algorithms in bandit-feedback scenarios.
We study mean-based algorithms when the time horizon is unknown and only bandit feedback is available. In this setting, we provide the first lower bound on the algorithm-defining sequence $Ξ³_t$ that formally establishes a limit on how fast these algorithms can learn. Additionally, we propose two mean-based algorithms: one generalizes $Ξ΅$-greedy, and the other extends the mean-based Exp3 to unknown horizons. Our experiments show that mean-based algorithms, although slightly slower, can perform competitively with other bandit-feedback algorithms.
We further analyze the relationship to no-regret algorithms. Depending on the choice of $Ξ³_t$, the intersection with no-regret algorithms is non-trivial, and we show that algorithms exist that are both mean-based and no-regret. This adds context to the "exploitability" of this class of algorithms that previous contributions suggest.