On-Policy and Off-Policy Learning for Large Action Spaces

2026-07-30

AI Summary

This work addresses the challenges of policy learning in contextual bandits with extremely large action spaces, where inefficient exploration, high variance of importance weights, and optimization difficulties commonly arise. To improve exploration efficiency in online settings, the authors propose two approaches—mixed-effects Thompson Sampling (meTS) and diffusion Thompson Sampling (dTS)—that explicitly model dependencies among actions. For offline settings, they introduce a latent-variable-based method, sDM, which integrates a differentiable pessimism mechanism with a concave policy-weighted log-likelihood objective to mitigate extrapolation bias and variance issues. Theoretical analysis yields regret bounds that scale with the effective number of actions, and empirical results demonstrate that the proposed methods significantly enhance both stability and performance of policy learning in large action spaces.

Abstract

This thesis studies policy learning in interactive systems where an agent observes a context, selects an action from a very large set, and receives partial feedback. The main framework is contextual bandits, with two paradigms: on-policy learning, where the agent interacts sequentially with the environment and minimizes regret, and off-policy learning, where it learns from logged data collected by a logging policy. In large action spaces, both settings face major challenges: inefficient exploration, sparse data coverage, high-variance importance weights, extrapolation bias, and difficult optimization landscapes. The first part develops structured Bayesian methods for on-policy learning. We introduce meTS, a mixed-effect extension of Thompson sampling, and dTS, which leverages diffusion-inspired priors to model dependencies between actions. These methods share information across actions and yield regret guarantees depending on an effective number of actions. The second part addresses off-policy learning. We propose sDM, a structured direct method based on latent variables, show that optimization error can dominate estimation error in large action spaces, and introduce concave, efficiently optimizable policy-weighted log-likelihood objectives. Finally, we develop differentiable pessimistic methods based on exponential smoothing and PAC-Bayesian bounds to control the bias-variance trade-off of regularized importance-sampling estimators.

Problem

• large action spaces
• on-policy learning
• off-policy learning
• contextual bandits
• policy learning

Innovation

• structured Bayesian methods
• large action spaces
• off-policy learning
• Thompson sampling
• pessimistic estimation