Scalable Policy Optimization for Networked Multi-Agent Reinforcement Learning with Continuous State-Action Spaces

2026-07-20

AI Summary

This work addresses the tension between local interaction and global coordination in large-scale networked multi-agent reinforcement learning within continuous state-action spaces by proposing the Continuous Distributed Coupled Policy Gradient (CDCPG) algorithm. Each agent maintains a localized policy based on a bounded graph neighborhood, approximates the local transition kernel using spectral random features, and optimizes a truncated action-value function. Theoretically, the paper establishes a localized Bellman equation to eliminate truncation bias, identifies a dimensionality barrier for temporal-difference stability under normalized random features, and introduces a verifiable persistent excitation condition. An adaptive neighborhood radius mechanism is designed to decouple computational and communication complexity from network size. Under an exponential spatial decay assumption, CDCPG achieves an $\tilde{O}(\varepsilon^{-2})$ sample complexity to optimize average per-agent stationarity within a provable error bound, with experiments confirming the efficacy of the locality structure and feature dimension selection.

Abstract

We develop the Continuous Distributed Coupled Policy Gradient (CDCPG) algorithm for cooperative reinforcement learning in networked Markov decision processes with continuous state and action spaces. Each agent maintains a local actor over a bounded graph neighborhood, and a localized least-squares temporal-difference critic evaluates a truncated action-value function through a spectral random-feature representation of the local transition kernel. The analysis makes four contributions. First, the truncated action-value function is constructed as a conditional expectation over the neighborhood, yielding a well-posed localized Bellman theory that removes the continuation-kernel mismatch of naive truncation arguments. Second, we expose a dimensional obstruction to temporal-difference stability for normalized random features and prove an unconditional excitation bound that reduces stability to a symmetric persistence-of-excitation condition, monitorable through an online matrix-concentration certificate. Third, under exponential spatial decay of agent interactions, the excitation condition, and smoothness of the objective, CDCPG drives an averaged per-agent stationarity measure to within any excess $ε$ of an explicitly characterized approximation floor using $\widetilde{\mathcal{O}}(ε^{-2})$ shared-oracle samples, and the excess dependence matches the smooth nonconvex first-order rate; per-agent computation and communication are governed by the neighborhood size rather than the network size. Fourth, an adaptive-locality rule selects the radius that balances truncation and graph-decay residuals against the target accuracy. Experiments on a networked linear-quadratic benchmark corroborate the locality and feature-dimension predictions.

Problem

• multi-agent reinforcement learning
• continuous state-action spaces
• scalable policy optimization
• networked systems
• locality

Innovation

• multi-agent reinforcement learning
• continuous state-action spaces
• localized policy gradient
• temporal-difference stability
• spectral random features