Gated Q-learning: Add Off-Policy Bias to Taste

2026-07-30

AI Summary

This work addresses the trade-off between off-policy bias and eligibility trace truncation in multi-step credit assignment within Q-learning. The authors propose an exploration-aware continuous gating mechanism that replaces conventional importance sampling, enabling adaptive interpolation between Watkins’s and Peng’s Q(λ) methods. This mechanism employs a state-action-dependent, differentiable gating function to dynamically modulate eligibility traces. Theoretically, the expected update operator under this approach is proven to be a contraction mapping with an exact fixed point, ensuring stability while allowing controlled balancing of bias and learning efficiency. Empirical results demonstrate that moderate gating effectively extends the credit assignment horizon and significantly accelerates early-stage learning, outperforming existing Q(λ) strategies that rely on extreme off-policy or on-policy assumptions.

Abstract

Multistep credit assignment is critical for sample-efficient reinforcement learning, yet managing off-policy bias in Q-learning remains a fundamental challenge. For 30 years, practitioners have been limited to a binary choice: eliminate the bias at the cost of severely truncated eligibility traces (Watkins' Q($λ$)), or ignore the bias to learn faster while injecting detrimental errors into the value estimates (Peng's Q($λ$)). Modern off-policy estimators fail to resolve this tension, as importance-sampling ratios collapse under Q-learning's greedy target policy. We introduce Gated Q-learning, a novel algorithmic framework that ends this dilemma by smoothly interpolating between the two historical extremes. Rather than relying on importance sampling, our approach employs a continuous, state-action-dependent gating mechanism to selectively attenuate eligibility traces in an exploration-aware manner. We provide a rigorous theoretical foundation for this mechanism, proving that the expected operator remains a contraction mapping and deriving its exact fixed point. Empirical evaluations verify that intermediate gating safely enables longer credit-assignment horizons, yielding faster initial learning than either extreme. Gated Q-learning offers a simple alternative to importance sampling while enabling customization of the effective multistep horizon and the amount of off-policy bias in Q-learning agents.

Problem

• off-policy bias
• Q-learning
• credit assignment
• eligibility traces
• reinforcement learning

Innovation

• Gated Q-learning
• off-policy bias
• eligibility traces
• credit assignment
• contraction mapping