GAWM: Global-Aware World Model for Multi-Agent Reinforcement Learning

2025-01-17

AI Summary

In partially observable multi-agent reinforcement learning (MARL), inefficient and non-convergent training arises from insufficient global state representation in world models and unstable pseudo-data distributions. To address these challenges within the centralized training with decentralized execution (CTDE) paradigm, we propose a centralized, globally aware world model. Our approach innovatively employs a Transformer architecture to fuse agents’ local observations into a unified, coherent representation of the global state. Furthermore, we introduce— for the first time in model-based MARL—a systematic dynamic distribution alignment mechanism to mitigate time-varying distributional shifts between pseudo-data and real data. Evaluated on the SMAC benchmark, our method achieves a 23% faster convergence rate, a 41% reduction in training variance, and a 3.2× improvement in sample efficiency, significantly outperforming state-of-the-art model-free and model-based baselines.

Technology Category

• Multiagent Systems: Adversarial Agents
• Machine Learning: Distributed Machine Learning & Federated Learning
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

In recent years, Model-based Multi-Agent Reinforcement Learning (MARL) has demonstrated significant advantages over model-free methods in terms of sample efficiency by using independent environment dynamics world models for data sample augmentation. However, without considering the limited sample size, these methods still lag behind model-free methods in terms of final convergence performance and stability. This is primarily due to the world model's insufficient and unstable representation of global states in partially observable environments. This limitation hampers the ability to ensure global consistency in the data samples and results in a time-varying and unstable distribution mismatch between the pseudo data samples generated by the world model and the real samples. This issue becomes particularly pronounced in more complex multi-agent environments. To address this challenge, we propose a model-based MARL method called GAWM, which enhances the centralized world model's ability to achieve globally unified and accurate representation of state information while adhering to the CTDE paradigm. GAWM uniquely leverages an additional Transformer architecture to fuse local observation information from different agents, thereby improving its ability to extract and represent global state information. This enhancement not only improves sample efficiency but also enhances training stability, leading to superior convergence performance, particularly in complex and challenging multi-agent environments. This advancement enables model-based methods to be effectively applied to more complex multi-agent environments. Experimental results demonstrate that GAWM outperforms various model-free and model-based approaches, achieving exceptional performance in the challenging domains of SMAC.

Problem

• Multi-Agent Reinforcement Learning
• Model-Based Methods
• Data Limitations and Incomplete Information

Innovation

• GAWM
• Transformer-based MARL
• Model-based Reinforcement Learning