Episodic Memory Temporal Consistency for Cooperative Multi-Agent Reinforcement Learning

2026-06-03

AI Summary

This work addresses the challenges of sparse rewards and difficult exploration in cooperative multi-agent reinforcement learning, which often lead to suboptimal policies and representation collapse. To mitigate these issues, the authors propose a novel approach that integrates temporally consistent semantic embeddings with a dynamic gating mechanism. Their method uniquely combines temporal-conditioned contrastive learning with Q-value overestimation suppression, selectively leveraging historical high-return trajectories to enhance both exploration efficiency and policy stability. Theoretical analysis provides error bounds that guarantee convergence properties. Empirical evaluation on the SMAC and Google Research Football (GRF) benchmarks demonstrates substantial improvements over state-of-the-art methods, achieving up to a 24% increase in win rate on the most challenging SMAC scenarios and a 28% average performance gain on GRF tasks.

Abstract

Cooperative Multi-Agent Reinforcement Learning (MARL) frequently suffers from severe reward sparsity and exploration bottlenecks. While episodic memory mechanisms mitigate these issues by reusing high-return trajectories, they often trap agents in local optima due to unconstrained incentive distribution and semantic representation collapse. To address this, we propose Episodic Memory Temporal Consistency (EMTC), a framework that robustly constructs and selectively leverages historical experiences. EMTC introduces two synergistic components: (1) a Temporally Consistent Semantic Embedder that integrates contrastive learning with time-conditioned state reconstruction, preventing representation collapse and enabling precise memory retrieval; and (2) a Temporal Consistency Gating Mechanism that dynamically modulates episodic incentives based on temporal consistency error. This adaptive gate filters misleading signals from pseudo-successful trajectories, effectively mitigating Q-value overestimation. We provide theoretical guarantees, establishing a strict error bound that directly links the observable temporal consistency error to the underlying trajectory optimality and representation quality. Extensive evaluations on the SMAC and GRF benchmarks demonstrate that EMTC consistently outperforms state-of-the-art baselines. Notably, compared to the strongest episodic baseline, EMTC achieves absolute win-rate improvements of up to 24% in super-hard SMAC scenarios and an average improvement of 28% across GRF tasks.

Problem

• Cooperative Multi-Agent Reinforcement Learning
• Episodic Memory
• Reward Sparsity
• Representation Collapse
• Exploration Bottleneck

Innovation

• Episodic Memory
• Temporal Consistency
• Multi-Agent Reinforcement Learning
• Contrastive Learning
• Q-value Overestimation