🤖 AI Summary
This work addresses the inefficiencies in coordination and insufficient policy robustness in cooperative multi-agent reinforcement learning (MARL) that arise from relying solely on either local or global perspectives. To overcome this limitation, the paper proposes a Hierarchical Leader-Critic (HLC) architecture inspired by team organizational structures. HLC introduces, for the first time, a multi-level perspective mechanism into MARL, enabling synergistic learning of local and global information without explicit inter-agent communication by integrating high-level objectives with low-level execution. Coupled with a sequential training strategy, the proposed method significantly outperforms single-level baselines across multiple cooperative MARL benchmarks and demonstrates superior scalability and robustness as the number of agents and task complexity increase.
📝 Abstract
Cooperative Multi-Agent Reinforcement Learning (MARL) solves complex tasks that require coordination from multiple agents, but is often limited to either local (independent learning) or global (centralized learning) perspectives. In this paper, we introduce a novel sequential training scheme and MARL architecture, which learns from multiple perspectives on different hierarchy levels. We propose the Hierarchical Lead Critic (HLC) - inspired by natural emerging distributions in team structures, where following high-level objectives combines with low-level execution. HLC demonstrates that introducing multiple hierarchies, leveraging local and global perspectives, can lead to improved performance with high sample efficiency and robust policies. Experimental results conducted on cooperative, non-communicative, and partially observable MARL benchmarks demonstrate that HLC outperforms single hierarchy baselines and scales robustly with increasing amounts of agents and difficulty.