Scalable Constrained Multi-Agent Reinforcement Learning via State Augmentation and Consensus for Separable Dynamics

2026-05-28

AI Summary

This work addresses the challenge of coordinating multi-agent systems under global resource constraints, where independent learning often fails to yield feasible solutions. Focusing on systems with separable dynamics, the paper proposes a decentralized reinforcement learning approach that integrates state-augmented policies with a lightweight consensus mechanism for dual variables exchanged among neighbors. This design enables local synchronization of Lagrange multipliers, ensuring global constraint satisfaction while preserving the scalability of independent agent training. The method overcomes the quadratic complexity growth inherent in conventional centralized training with decentralized execution (CTDE) frameworks. Evaluated on a smart grid demand response task, it scales effectively to thousands of agents—satisfying capacity constraints and achieving successful scheduling—whereas CTDE baselines are limited to tens of agents and frequently produce infeasible outcomes.

Abstract

We present a distributed approach for constrained Multi-Agent Reinforcement Learning (MARL) that combines state-augmented policy learning with distributed consensus over dual variables. Our method targets systems where agents have separable dynamics but must coordinate to satisfy global resource constraints, a setting in which, as we demonstrate empirically, independent learning fails to produce feasible solutions because agents cannot determine appropriate individual contributions toward collective constraint satisfaction. The key technical contribution is showing that lightweight neighbor-to-neighbor consensus over Lagrange multipliers suffices for globally coordinated constraint enforcement while preserving the scalability of independent training. Each agent learns a single augmented policy offline, conditioned on both its local state and a dual variable encoding constraint feedback. During execution, agents reach agreement on this dual variable through local communication alone. We prove that under mild connectivity assumptions, the consensus error among agents' multipliers is bounded, and show that this translates to a bounded constraint violation that decreases with graph connectivity and the number of consensus rounds. Unlike centralized training with decentralized execution (CTDE) approaches, whose complexity grows at least quadratically with agent count, our method scales linearly in both training and execution. Experiments on smart grid demand response demonstrate that consensus coordination is \emph{essential for feasibility}: without it, agents satisfy grid capacity constraints only by indefinitely postponing demand, a degenerate non-solution. With consensus, agents converge to a shared dual variable and satisfy both grid constraints and demand fulfillment, scaling to thousands of agents while CTDE baselines are limited to dozens.

Problem

• Constrained Multi-Agent Reinforcement Learning
• Global Resource Constraints
• Separable Dynamics
• Feasibility
• Scalability

Innovation

• Constrained Multi-Agent Reinforcement Learning
• State Augmentation
• Distributed Consensus
• Lagrange Multipliers
• Scalable MARL