Online KL-Regularized Reinforcement Learning with Function Approximation under Misspecification

2026-06-04

AI Summary

This work addresses the lack of theoretical guarantees for KL-regularized contextual bandits and episodic reinforcement learning with function approximation under model misspecification. The authors propose a novel framework that introduces a KL-based misspecification notion to formally characterize non-realizability, integrating regression-oracle-based algorithms with Gibbs policy updates for online learning. Building on this framework, they establish a high-probability KL-regret bound that explicitly accounts for model misspecification. This bound not only recovers the standard realizable case as a special instance but also provides the first rigorous theoretical guarantee for algorithms operating under imperfect modeling assumptions. Consequently, the approach significantly enhances the applicability and robustness of KL-regularized methods in function approximation settings.

Abstract

We study KL-regularized contextual bandits and episodic reinforcement learning (RL) under general function approximation with model misspecification. Existing guarantees rely on realizability and therefore do not extend to misspecified models, where classical regret bounds may fail. This work introduces KL misspecification formulations for contextual bandits and episodic RL and analyzes regression-based algorithms with Gibbs policy updates. High-probability KL-regret guarantees with explicit misspecification terms are established, recovering the standard realizable KL-regularized setting as a special case.

Problem

• KL-regularized
• reinforcement learning
• model misspecification
• function approximation
• contextual bandits

Innovation

• KL regularization
• model misspecification
• function approximation
• contextual bandits
• episodic reinforcement learning