Variance-Reduced Q-Learning over Static and Time-Varying Networks

2026-07-23

AI Summary

This work addresses the challenge of enabling multiple agents to collaboratively learn an optimal action-value function in a shared Markov decision process while minimizing communication overhead. The paper proposes VRDQ, a round-based distributed Q-learning algorithm that integrates variance reduction, consensus protocols, and a distributed reinforcement learning framework. Within each round, agents locally estimate the Bellman optimality operator and disseminate information via consensus updates. VRDQ achieves linear speedup in sample complexity—the first method to do so—while reducing communication complexity to merely Õ(1), substantially outperforming existing approaches. Moreover, it guarantees high-probability finite-time convergence under both static and time-varying network topologies, thereby realizing the full benefit of collaborative linear acceleration.

Abstract

We investigate a decentralized reinforcement learning problem involving multiple agents that interact with the same Markov Decision Process (MDP). The agents can exchange information over a network to collectively learn the optimal state-action value function. For this setting, we introduce a novel epoch-based distributed $Q$-learning algorithm called VRDQ, where within each epoch, agents locally estimate the Bellman optimality operator and diffuse information using a consensus-based protocol. For both static and time-varying networks, we establish high-probability finite-time convergence rates for VRDQ that enjoy linear speedups from collaboration. Crucially, we prove that such speedups in sample-complexity require only $\tilde{O}(1)$ communication, substantially improving upon the communication costs in prior work.

Problem

• Decentralized Reinforcement Learning
• Multi-agent Q-Learning
• Markov Decision Process
• Networked Collaboration
• Sample Complexity

Innovation

• variance-reduced Q-learning
• decentralized reinforcement learning
• finite-time convergence
• linear speedup
• communication efficiency