A Simple and Adaptive Learning Rate for FTRL in Online Learning with Minimax Regret of θ(T2/3) and its Application to Best-of-Both-Worlds

2024-05-30

Neural Information Processing Systems

AI Summary

Existing adaptive learning rate methods are scarce for online learning problems with minimax regret of order Θ(T²⁄₃), such as partial monitoring, graph bandits, and costly-observation multi-armed bandits. Method: We propose the first simple, adaptive learning rate framework tailored to this regret scale, built upon Follow-the-Regularized-Leader (FTRL) with Tsallis entropy regularization. Our design achieves unified optimization in both stochastic and adversarial environments by precisely balancing stability, penalty, and bias terms. Contribution/Results: Unlike existing Best-of-Both-Worlds algorithms relying on intricate rate constructions, our approach significantly simplifies algorithm design while attaining tighter regret upper bounds across all three Θ(T²⁄₃) problem classes. It is the first method to simultaneously improve performance in both stochastic and adversarial settings. The resulting learning rate depends only on logarithmic-scale terms, ensuring both theoretical tightness and practical applicability.

Technology Category

Machine Learning: Online Learning & BanditsGame Theory and Economic Paradigms: Adversarial LearningSearch and Optimization: Learning to Search

Application Category

Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for rankingGraph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphsUser Modeling, Personalization and Recommendation: User modeling for targeted and personalized online advertising

Abstract

Follow-the-Regularized-Leader (FTRL) is a powerful framework for various online learning problems. By designing its regularizer and learning rate to be adaptive to past observations, FTRL is known to work adaptively to various properties of an underlying environment. However, most existing adaptive learning rates are for online learning problems with a minimax regret of $Theta(sqrt{T})$ for the number of rounds $T$, and there are only a few studies on adaptive learning rates for problems with a minimax regret of $Theta(T^{2/3})$, which include several important problems dealing with indirect feedback. To address this limitation, we establish a new adaptive learning rate framework for problems with a minimax regret of $Theta(T^{2/3})$. Our learning rate is designed by matching the stability, penalty, and bias terms that naturally appear in regret upper bounds for problems with a minimax regret of $Theta(T^{2/3})$. As applications of this framework, we consider three major problems with a minimax regret of $Theta(T^{2/3})$: partial monitoring, graph bandits, and multi-armed bandits with paid observations. We show that FTRL with our learning rate and the Tsallis entropy regularizer improves existing Best-of-Both-Worlds (BOBW) regret upper bounds, which achieve simultaneous optimality in the stochastic and adversarial regimes. The resulting learning rate is surprisingly simple compared to the existing learning rates for BOBW algorithms for problems with a minimax regret of $Theta(T^{2/3})$.

Problem

• Develops adaptive learning rate for FTRL in online learning.
• Addresses minimax regret of Θ(T^2/3) with indirect feedback.
• Improves Best-of-Both-Worlds regret bounds across regimes.

Innovation

• Adaptive learning rate for FTRL
• Minimax regret of Θ(T^(2/3))
• Tsallis entropy regularizer application