Regularized Offline Policy Optimization with Posterior Hybrid Bayesian Belief

2026-05-30

AI Summary

This work addresses the challenge in offline reinforcement learning of jointly quantifying and optimizing epistemic uncertainty arising from limited data coverage and ambiguity in dynamics model identification. The authors propose the Posterior Hybrid Bayesian (PhyB) framework, which treats the dynamics model as a random variable and maintains a posterior belief over it. By constructing convex combinations over subsets of models to approximate the expected objective, PhyB enables efficient policy optimization without resorting to computationally intractable search or strong posterior assumptions. The approach preserves the adaptability of Bayesian RL while providing a metric-agnostic guarantee of monotonic performance improvement. Empirical results demonstrate that PhyB achieves state-of-the-art performance across multiple offline RL benchmarks, confirming its effectiveness and robustness.

Abstract

Offline reinforcement learning (RL) aims to optimize policies from pre-collected datasets. A bottleneck of this paradigm is managing epistemic uncertainty, which arises from limited data coverage (sample-level) and the ambiguity in identifying transition dynamics from finite data (model-level). To provide a unified quantification of these uncertainties, Bayesian RL has been proposed by treating the dynamics model as a random variable and maintaining a corresponding belief. Despite its theoretical appeal, policy optimization in Bayesian RL remains computationally challenging as it requires solving composite objectives with expectations. Prior methods either employ search-based techniques with poor computational scalability or impose restrictive posterior assumptions that sacrifice the adaptability of Bayesian RL. To address these limitations, we propose Posterior Hybrid Bayesian Belief (PhyB), which reformulates the expectation as a convex combination over a subset of dynamics models. Theoretical analysis demonstrates that the objective discrepancy induced by this approximation remains bounded. Based on PhyB, we develop an iterative regularized policy optimization algorithm that provides metric-agnostic guarantees for monotonic improvement until convergence. Empirical results demonstrate that PhyB achieves state-of-the-art performance on various benchmarks.

Problem

• offline reinforcement learning
• epistemic uncertainty
• Bayesian RL
• policy optimization
• transition dynamics

Innovation

• Bayesian Reinforcement Learning
• Offline Policy Optimization
• Epistemic Uncertainty
• Convex Combination
• Regularized Optimization