🤖 AI Summary
This work addresses the scalability bottleneck in multi-agent reinforcement learning caused by the linear growth of input dimensionality with the number of agents in centralized critics. To overcome this limitation, the authors propose MADDPG-K, which introduces a k-nearest-neighbor mechanism based on Euclidean distance into the MADDPG framework for the first time. This allows each critic to observe only a fixed number of neighboring agents, thereby maintaining constant input dimensionality. While preserving the centralized training with decentralized execution paradigm, MADDPG-K significantly reduces computational complexity. Experimental results demonstrate that MADDPG-K achieves performance comparable to or better than MADDPG in multi-particle cooperative and competitive tasks, with faster convergence and markedly improved computational efficiency and scalability as the number of agents increases.
📝 Abstract
We propose MADDPG-K, a scalable extension to Multi-Agent Deep Deterministic Policy Gradient (MADDPG) that addresses the computational limitations of centralized critic approaches. Centralized critics, which condition on the observations and actions of all agents, have demonstrated significant performance gains in cooperative and competitive multi-agent settings. However, their critic networks grow linearly in input size with the number of agents, making them increasingly expensive to train at scale. MADDPG-K mitigates this by restricting each agent's critic to the $k$ closest agents under a chosen metric which in our case is Euclidean distance. This ensures a constant-size critic input regardless of the total agent count. We analyze the complexity of this approach, showing that the quadratic cost it retains arises from cheap scalar distance computations rather than the expensive neural network matrix multiplications that bottleneck standard MADDPG. We validate our method empirically across cooperative and adversarial environments from the Multi-Particle Environment suite, demonstrating competitive or superior performance compared to MADDPG, faster convergence in cooperative settings, and better runtime scaling as the number of agents grows. Our code is available at https://github.com/TimGop/MADDPG-K .