Geometrically Averaged Hard Target Updates for Linear Q-Learning

2026-06-09

AI Summary

This work addresses the instability of target updates in linear Q-learning by introducing a λ-target update mechanism that smooths periodic hard targets through geometrically weighted averaging, thereby establishing a continuous spectrum of update strategies lying between one-step updates and projected Q-value iteration. The proposed approach unifies existing extreme update schemes and provides a theoretical analysis framework grounded in switched system models. In deterministic environments, the λ-target mechanism significantly enhances training stability, offering both a theoretical foundation and a scalable methodology for stable target updates in stochastic reinforcement learning settings.

Abstract

Periodic hard target updates are among the most common stabilization devices in modern deep Q-learning. Recent studies suggest that target updates can improve stability in Q-learning with function approximation, including linear function approximation. We introduce and analyze the so-called $λ$-target update, obtained by averaging the $m$-periodic target update maps with $λ$-geometric weights $(1-λ)λ^{m-1}$, $λ\in [0,1]$. The endpoint $λ=0$ recovers the one-period target update, while the continuous endpoint $λ\uparrow1$ recovers projected Q-value iteration. We study this mechanism for Q-learning with linear function approximation, namely linear Q-learning, using a switching-system model and related tools. For clarity, the paper treats a deterministic version; the formulation extends to stochastic reinforcement-learning settings.

Problem

• Q-learning
• linear function approximation
• target updates
• stability
• reinforcement learning

Innovation

• geometric averaging
• hard target update
• linear Q-learning
• switching system
• function approximation