Revisiting TD Target Aggregation under Uncertainty in Q-Learning

2026-08-03

AI Summary

This work addresses the susceptibility of Q-learning to noise in the presence of uncertainty in Q-value estimates, which often leads to overestimation and training instability. To mitigate this issue without altering the core Q-learning framework, the authors propose SADQ, a method that incorporates a learned dynamics model to perform one-step rollout predictions and restructures the temporal difference (TD) target aggregation process. By employing a hybrid Bellman update mechanism, SADQ selectively suppresses unreliable maximized Q-value estimates while preserving the desirable fixed point. This approach enables structured action-value aggregation and demonstrates significantly improved training stability and performance across a range of domains, including classical control tasks, real-world robotic environments, and the Atari benchmark, outperforming several strong DQN variants.

Abstract

Deep Q-Networks (DQNs) learn value functions through bootstrapped temporal-difference updates, where future returns are approximated using a greedy maximization over next-state action values. While effective, this aggregation rule is inherently sensitive to estimation noise: when Q-values are uncertain, the maximization operator deterministically favors the largest estimate, regardless of its reliability, leading to amplified errors through bootstrapping. In this work, we propose the \textbf{S}uccessor Rollout \textbf{A}ggregation \textbf{D}eep \textbf{Q}-Network (SADQ), a simple modification to Q-learning that regularizes how the TD target is formed. SADQ uses one-step rollout predictions from a learned dynamics model to guide the comparison among candidate next-state actions, introducing additional structure into the aggregation step without altering the underlying learning framework. The resulting mixed Bellman update attenuates unreliable maxima while preserving the standard fixed point under diminishing model error. We provide theoretical analysis showing that SADQ reduces bootstrap-induced overestimation in a pointwise manner. Empirically, SADQ consistently improves training stability across classical control tasks, real-world vector-based environments, and Atari benchmarks when compared to strong DQN variants.

Problem

• Q-learning
• temporal-difference learning
• overestimation
• uncertainty
• bootstrapping

Innovation

• SADQ
• TD target aggregation
• overestimation bias
• model-based rollout
• Q-learning