Distributed Value Decomposition Networks with Networked Agents

2025-02-11

AI Summary

This work addresses decentralized multi-agent reinforcement learning (MARL) under partial observability without centralized training, aiming to enable cooperative optimization of the joint cumulative reward by heterogeneous or homogeneous agents relying solely on local observations and peer-to-peer communication. To this end, we propose DVDN and its enhanced variant DVDN(GT), the first algorithms to achieve fully decentralized value decomposition. DVDN employs locally parameterized Q-functions, a shared target consistency constraint, and a local target estimation mechanism to align each agent’s local Q-value with the global Q-value, thereby mitigating information loss induced by limited communication. Evaluated across ten tasks from three standard MARL benchmarks, DVDN(GT) achieves performance comparable to centralized-training VDN while significantly improving effectiveness and robustness in communication-constrained settings.

Technology Category

• Multiagent Systems: Multiagent Learning
• Machine Learning: Distributed Machine Learning & Federated Learning
• Search and Optimization: Distributed Search

Application Category

• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• Systems and Infrastructure for Web, Mobile and WoT: Decentralized Web and Fediverse systems

Abstract

We investigate the problem of distributed training under partial observability, whereby cooperative multi-agent reinforcement learning agents (MARL) maximize the expected cumulative joint reward. We propose distributed value decomposition networks (DVDN) that generate a joint Q-function that factorizes into agent-wise Q-functions. Whereas the original value decomposition networks rely on centralized training, our approach is suitable for domains where centralized training is not possible and agents must learn by interacting with the physical environment in a decentralized manner while communicating with their peers. DVDN overcomes the need for centralized training by locally estimating the shared objective. We contribute with two innovative algorithms, DVDN and DVDN (GT), for the heterogeneous and homogeneous agents settings respectively. Empirically, both algorithms approximate the performance of value decomposition networks, in spite of the information loss during communication, as demonstrated in ten MARL tasks in three standard environments.

Problem

• Distributed training under partial observability
• Decentralized multi-agent reinforcement learning
• Local estimation of shared objective

Innovation

• Distributed Value Decomposition Networks
• Decentralized multi-agent training
• Local shared objective estimation