Continual Graph Multi-Agent Reinforcement Learning

2026-10-07

AI Summary

This study addresses the challenges of insufficient transfer and catastrophic forgetting in multi-agent reinforcement learning (MARL) arising from the lack of mechanisms to exploit task structure. To this end, it proposes CGMARL, the first graph-based continual MARL framework. CGMARL models tasks as attributed graphs and introduces a frozen graph encoder method, FROG, to preserve historical structural knowledge and mitigate forgetting. Additionally, a dedicated evaluation benchmark, GRAFO, is constructed. Experimental results demonstrate that the proposed approach significantly outperforms existing baselines across multiple scenarios, effectively enhancing cross-task transfer performance and resilience against forgetting.

Abstract

In Continual Multi-Agent Reinforcement Learning (CMARL), agents learn cooperative policies across sequences of tasks, aiming to adapt effectively to new tasks while preserving the ability to solve previously encountered ones. In many applications, tasks differ in their underlying structure, which can represent, for example, distinct operational conditions or target configurations (e.g., different network topologies in power grids or arrangements in formation control). Existing CMARL methods lack dedicated mechanisms to leverage this structural information when learning new tasks, failing to promote transfer and mitigate forgetting. To fill this gap, we propose Continual Graph Multi-Agent Reinforcement Learning (CGMARL), a novel framework for CMARL problems in which task sequences are mapped into a series of attributed graphs, each modeling a task-specific structure. In CGMARL, each graph determines the environment dynamics (next states and/or rewards) and the number of agents for the corresponding task. Then, we present Graph-based Formation (GRAFO), the first CGMARL benchmark, and show how forgetting arises in this setting. Finally, to address this limitation, we propose Frozen Graph Encoder (FROG), a method that relies on a frozen graph backbone to preserve past structural information in graph-based CMARL policies. Experiments on GRAFO show that pairing FROG with existing CL methods substantially improves performance on multiple CGMARL scenarios.

Problem

• Continual Multi-Agent Reinforcement Learning
• Graph Structure
• Catastrophic Forgetting
• Knowledge Transfer

Innovation

• Continual Multi-Agent Reinforcement Learning
• Graph Neural Networks
• Catastrophic Forgetting
• Transfer Learning
• Benchmark