🤖 AI Summary
This work addresses the challenge of Q-value overestimation in offline reinforcement learning, which arises from distributional shift between the behavior policy and the learning policy and can mislead policy optimization. To mitigate this issue, the paper proposes Behavior-Advantage Corrected Policy Evaluation (BAC-PE), a novel approach that leverages the Q-function of the behavior policy to correct the Q-estimates of the learning policy. BAC-PE further integrates a diffusion model for policy representation, enhancing expressiveness and training stability through Q-guided sampling and distribution-matching regularization. Theoretical analysis establishes the convergence of BAC-PE and provides an upper bound on its estimation error. Empirical evaluation on the D4RL benchmark demonstrates that the proposed method significantly outperforms existing offline reinforcement learning algorithms, achieving state-of-the-art performance.
📝 Abstract
In offline reinforcement learning (RL), the distribution shift between behavioral data and the learned policy can lead to erroneous \emph{Q}-value estimation, thereby misguiding the direction of policy optimization. To address this issue, we develop a behavioral advantage corrected policy evaluation (BAC-PE) approach, which utilizes the \emph{Q}-function of the behavior policy to correct the learned policy's \emph{Q}-function, thus mitigating pessimistic conservatism and overestimation bias. Furthermore, the convergence of BAC-PE is analyzed theoretically, and an upper bound on the difference between the learned \emph{Q}-function and the true \emph{Q}-function is derived. To alleviate distribution shift, this work employs diffusion models to represent both the behavior policy and the learned policy, performing distribution matching for accurate policy regularization. Additionally, \emph{Q}-value guidance is incorporated into the training process to achieve effective policy improvement. By combining BAC-PE with diffusion policy modeling, we propose the diffusion policy with behavioral advantage correction (DPBAC) algorithm. Compared to existing offline methods, DPBAC demonstrates stronger policy representation capabilities and effectively mitigates the bias in \emph{Q}-value estimation. Experimental results on multiple domains of D4RL tasks show that DPBAC achieves superior performance, with notable advantages over state-of-the-art (SOTA) algorithms.