A Unified Algorithmic Framework for Hybrid Reinforcement Learning in Tabular MDPs with Shifted Transition Dynamics

2026-07-27

AI Summary

This work addresses the challenge of effectively integrating offline data from a source environment—subject to transition dynamics shift—with online interactions in a target environment within hybrid reinforcement learning. The paper proposes a unified algorithmic framework that enables efficient policy learning in tabular MDPs. Its key contribution lies in establishing, for the first time, matching upper and lower theoretical bounds for this hybrid setting with dynamic shifts, alongside a fine-grained bias-aware mechanism for data utilization. Specifically, the proposed MIN-UCB-VI and MAX-LCB-VI algorithms combine confidence intervals with bias-correction techniques to seamlessly fuse offline and online data. Theoretical analysis yields both instance-dependent and instance-independent regret bounds, as well as upper bounds on suboptimality gaps, while empirical results corroborate the algorithms’ efficacy and near-optimality.

Abstract

This paper investigates a hybrid reinforcement learning setting in tabular Markov Decision Processes (MDPs), where an agent aims to learn an optimal policy by combining online interactions with a target environment and offline data from a source environment. A central challenge is that offline data may be collected from outdated environments with shifted transition dynamics, making naive integration of historical data ineffective. To address this, we propose a unified algorithmic framework featuring two algorithms: MIN-UCB-VI for regret minimization and MAX-LCB-VI for best policy identification. Both algorithms leverage fine-grained bias information to more effectively exploit offline data under general transition shifts. We provide theoretical guarantees for our framework, including both instance-dependent and independent upper bounds on regret and sub-optimality gap. Furthermore, we establish matching lower bounds to demonstrate the optimality of our approach and validate our theoretical findings through extensive experiments.

Problem

• hybrid reinforcement learning
• tabular MDPs
• shifted transition dynamics
• offline data
• policy learning

Innovation

• hybrid reinforcement learning
• shifted transition dynamics
• unified algorithmic framework
• offline-online integration
• instance-dependent bounds