Collaborative Weighting with Pessimistic Critic for Mitigating Overestimation in Off-Policy Reinforcement Learning

2026-07-29

AI Summary

Off-policy reinforcement learning often suffers from value overestimation and training instability due to noisy temporal difference targets and greedy updates. This work proposes the Collaborative Weighted Actor-Critic (CWAC) framework, which uniquely integrates uncertainty modeling of return distributions with a collaborative weighting mechanism and introduces a sampling-based stochastic pessimistic value estimation to effectively suppress the influence of unreliable samples, thereby mitigating error propagation and overestimation bias. The method is compatible with mainstream algorithms such as SAC, TD3, and DDPG, achieving significant improvements in both performance and stability across a variety of continuous control tasks while incurring minimal computational overhead, allowing seamless integration into existing off-policy frameworks.

Abstract

Deep off-policy reinforcement learning algorithms for continuous control typically rely on neural value function approximation to guide policy improvement. However, temporal-difference (TD) learning introduces noisy targets, resulting in non-stationary optimization, while greedy policy updates amplify early-stage estimation errors. The recursive propagation of such errors leads to persistent overestimation bias and degraded training stability in actor-critic methods. Existing approaches attempt to alleviate this issue via prioritized sampling or modified value learning objectives, but often overemphasize high-uncertainty transitions caused by limited data coverage or bootstrapping errors, thereby further amplifying bias.In this paper, we propose Collaborative Weighting Actor-Critic (CWAC), a unified framework that explicitly accounts for predictive uncertainty in value estimation. CWAC employs distributional critic to model return uncertainty and introduces a collaborative weighting mechanism that jointly reweights TD-errors and uncertainty, enabling robust learning from reliable samples while suppressing noisy updates. In addition, we incorporate a stochastic pessimistic value estimation scheme via sampling from the return distribution, which effectively mitigates error propagation during policy improvement. CWAC can be seamlessly integrated into existing off-policy algorithm frameworks such as SAC, TD3, and DDPG with minimal overhead. Empirical results demonstrate that our proposed method significantly enhances performance across a diverse range of simulated tasks. Our code is publicly available at https://anonymous.4open.science/r/CWAC-348E.

Problem

• overestimation
• off-policy reinforcement learning
• temporal-difference learning
• value estimation bias
• actor-critic methods

Innovation

• distributional critic
• collaborative weighting
• pessimistic value estimation
• overestimation bias
• off-policy reinforcement learning