Dreamer-CPC: Message Learning with World Models for Decentralized Multi-agent Reinforcement Learning

2026-07-22

AI Summary

This work addresses the limitation of existing communication methods in partially observable multi-agent reinforcement learning, which rely solely on current observations and thus struggle to convey historical information, impeding effective coordination. To overcome this, the paper proposes Dreamer-CPC, the first approach that leverages the temporal latent states of a world model for communication. By integrating Collective Predictive Coding (CPC) into the DreamerV3 framework, agents infer and exchange messages derived from latent states that encode histories of past observations and actions, enabling efficient decentralized communication. This paradigm shifts beyond conventional instantaneous-observation-based messaging and demonstrates substantial performance gains over both IPPO-CPC and non-communicating baselines—achieving 4–5 times higher episode returns—on the Observer benchmark and a newly introduced CatchApple task, thereby validating its superior coordination capability under observation scarcity.

Abstract

In multi-agent reinforcement learning (MARL), inter-agent communication is effective for improving performance under partial observability. Representation learning-based approaches enable decentralized agents to learn messages grounded in their own observations, but they rely only on current observations and cannot convey information accumulated over time. We propose Dreamer-CPC, a decentralized model-based MARL method that integrates message learning based on Collective Predictive Coding (CPC) into the world model of DreamerV3. Each agent independently maintains a world model and a message module, and infers and exchanges messages from the latent states of the world model that reflect the history of past observations and actions. We evaluated Dreamer-CPC in two environments: Observer, a non-cooperative information-sharing task, and CatchApple, a newly introduced task in which task-relevant observations are temporarily missing. In both environments, Dreamer-CPC outperformed IPPO-CPC, an existing CPC-based method that generates messages from current observations, as well as no-communication baselines. In particular, in CatchApple, Dreamer-CPC achieved 4 to 5 times the episode return of IPPO-CPC, demonstrating effective coordination where other methods fail due to missing observations. These results suggest that communication grounded in the latent dynamics of world models can support decentralized decision-making when current observations alone are insufficient.

Problem

• multi-agent reinforcement learning
• partial observability
• inter-agent communication
• message learning
• world models

Innovation

• world models
• multi-agent reinforcement learning
• Collective Predictive Coding
• decentralized communication
• latent dynamics