Networked Agents in the Dark: Team Value Learning under Partial Observability

2025-01-15

AI Summary

We address the problem of collaborative decision-making in networked multi-agent systems under partial observability, communication constraints, and privacy requirements. To this end, we propose a Networked Dynamic Partially Observable Markov Game (ND-POMG) modeling framework and the DNA-MARL distributed algorithm. DNA-MARL operates solely on local observations and sparse, unreliable communication—supporting time-varying topologies and packet loss—and employs consensus-based distributed approximation of the team value function. It integrates local policy gradient optimization with individual reward learning, eliminating the need for global state information or joint observations. Evaluated on standard MARL benchmarks, DNA-MARL significantly outperforms existing methods. Moreover, it demonstrates superior robustness and practical applicability in privacy-sensitive, bandwidth-constrained real-world edge computing scenarios.

Technology Category

• Multiagent Systems: Mechanism Design
• Machine Learning: Distributed Machine Learning & Federated Learning
• Planning, Routing, and Scheduling: Planning with Markov Models (MDPs, POMDPs)

Application Category

• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Federated Web and WoT systems, including distributed, federated and edge-based data processing

Abstract

We propose a novel cooperative multi-agent reinforcement learning (MARL) approach for networked agents. In contrast to previous methods that rely on complete state information or joint observations, our agents must learn how to reach shared objectives under partial observability. During training, they collect individual rewards and approximate a team value function through local communication, resulting in cooperative behavior. To describe our problem, we introduce the networked dynamic partially observable Markov game framework, where agents communicate over a switching topology communication network. Our distributed method, DNA-MARL, uses a consensus mechanism for local communication and gradient descent for local computation. DNA-MARL increases the range of the possible applications of networked agents, being well-suited for real world domains that impose privacy and where the messages may not reach their recipients. We evaluate DNA-MARL across benchmark MARL scenarios. Our results highlight the superior performance of DNA-MARL over previous methods.

Problem

• Multi-agent Learning
• Privacy Preservation
• Information Transmission Limitations

Innovation

• DNA-MARL
• Multi-agent Collaboration
• Privacy-preserving Communication