Non-Stationary Latent Auto-Regressive Bandits

2024-02-05

arXiv.org

AI Summary

This work studies the latent-state-driven non-stationary multi-armed bandit problem, where reward means evolve according to an underlying linear autoregressive (AR) dynamic—without requiring a pre-specified non-stationarity budget. We formulate the problem as a linear dynamical system and propose LARL (Latent Autoregressive Learning), a novel algorithm that jointly performs implicit latent-variable inference, approximate steady-state Kalman filtering, and online system parameter estimation to achieve interpretable non-stationarity adaptation. Theoretically, LARL achieves sublinear regret when observation noise variance is small; its regret bound explicitly depends on AR coefficients and observation noise variance, yielding strong interpretability. Empirically, LARL significantly outperforms state-of-the-art non-stationary bandit baselines across diverse benchmarks.

Technology Category

• Machine Learning: Online Learning & Bandits
• Multiagent Systems: Multiagent Learning
• Intelligent Robots: State Estimation

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

For the non-stationary multi-armed bandit (MAB) problem, many existing methods allow a general mechanism for the non-stationarity, but rely on a budget for the non-stationarity that is sub-linear to the total number of time steps $T$. In many real-world settings, however, the mechanism for the non-stationarity can be modeled, but there is no budget for the non-stationarity. We instead consider the non-stationary bandit problem where the reward means change due to a latent, auto-regressive (AR) state. We develop Latent AR LinUCB (LARL), an online linear contextual bandit algorithm that does not rely on the non-stationary budget, but instead forms good predictions of reward means by implicitly predicting the latent state. The key idea is to reduce the problem to a linear dynamical system which can be solved as a linear contextual bandit. In fact, LARL approximates a steady-state Kalman filter and efficiently learns system parameters online. We provide an interpretable regret bound for LARL with respect to the level of non-stationarity in the environment. LARL achieves sub-linear regret in this setting if the noise variance of the latent state process is sufficiently small with respect to $T$. Empirically, LARL outperforms various baseline methods in this non-stationary bandit problem.

Problem

• Addresses non-stationary multi-armed bandit problem with latent auto-regressive state.
• Develops LARL algorithm for online linear contextual bandits without non-stationary budget.
• Provides sub-linear regret bound for LARL under small latent state noise variance.

Innovation

• Uses latent auto-regressive state modeling
• Implements online linear contextual bandit algorithm
• Approximates steady-state Kalman filter efficiently