Expert Behavior Prior Reinforcement Learning

2026-07-23

AI Summary

This work addresses the limitations of existing behavioral prior methods in reinforcement learning, which rely on static offline datasets suffering from low diversity and poor trajectory quality, leading to inefficient exploration and unstable training. To overcome these issues, the paper proposes Expert Behavioral Prior (EBP), a novel algorithm that abandons the conventional offline pre-training paradigm and instead dynamically generates high-quality expert action priors from an online replay buffer. EBP integrates three synergistic components: a Q-guided Conditional Variational Autoencoder (Q-CVAE), Expert Policy Guidance (EPG), and Policy Gradient Correction (PGC), enabling efficient and stable online policy optimization. Experimental results demonstrate that EBP significantly outperforms state-of-the-art methods across Gym, PyBullet, and DMControl benchmarks, achieving superior sample efficiency and convergence stability.

Abstract

Behavior prior reinforcement learning (BPRL) has emerged as a promising paradigm to improve sample efficiency in online reinforcement learning (RL) by leveraging policy priors derived from offline demonstrations. However, most existing BPRL methods rely on static offline datasets, which often suffer from low data diversity and suboptimal trajectory quality. This reliance restricts the effectiveness of policy priors, hindering both policy exploitation and stability during online training. Consequently, agents are prone to inefficient exploration and unstable learning dynamics. To address these limitations, we deviate from existing offline pre-training methods and propose an Expert Behavior Prior (EBP) algorithm. Specifically, we introduce a Q-guided conditional variational autoencoder (Q-CVAE) that learns to generate expert policy priors directly from the online replay buffer. This enables the generation of high-value actions for guiding policy updates without relying on pre-collected expert trajectories. To further enhance policy exploitation, we propose an expert policy guidance (EPG) mechanism that selects expert actions from a generative support set, and we integrate a policy gradient correction (PGC) module to harmonize Q-guidance with expert supervision, promoting stable and consistent policy improvement. Extensive experiments conducted on robotic control (Gym, PyBullet) and industrial control (DMControl) benchmarks demonstrate that EBP significantly outperforms state-of-the-art online RL algorithms, achieving higher sample efficiency and more stable convergence.

Problem

• behavior prior reinforcement learning
• offline demonstrations
• sample efficiency
• policy stability
• exploration inefficiency

Innovation

• Expert Behavior Prior
• Q-guided CVAE
• Online Reinforcement Learning
• Policy Gradient Correction
• Sample Efficiency