HySTAR: Anchored Hypergraphs for Stable Credit Assignment in Cooperative Multi-Agent Reinforcement Learning

2026-09-25

AI Summary

This study addresses structural objective drift and credit assignment instability caused by dynamic topologies in cooperative multi-agent reinforcement learning. Building upon the MAPPO framework, it introduces the concept of "structural objective drift" for the first time. Methodologically, an anchored overlapping sparse hypergraph serves as a fixed decomposition base, while a spatiotemporal encoder decouples adaptive representations from the value decomposition structure. Additionally, a spatiotemporally correlated advantage function is designed to balance topological consistency with interaction adaptability. Experimental evaluations on SMAC and GRF benchmarks demonstrate that the proposed method significantly outperforms mainstream algorithms, achieving a 16.7% win-rate improvement in the most challenging SMAC scenarios and accelerating convergence by 40.2% in traffic intersection tasks.

Abstract

Cooperative multi-agent reinforcement learning under partial observability and shared rewards requires assigning team outcomes to individual agents and high-order coalitions. A MAPPO-style critic compresses joint behavior into one global value, while critics that dynamically reconstruct the grouping topology change the mapping from agents and coalitions to value components as interactions or active agents evolve. We refer to this inconsistency as structural target drift. We introduce HySTAR, a MAPPO-based framework that separates adaptive representation learning from a temporally consistent high-order value-decomposition basis. HySTAR anchors an overlapping sparse hypergraph as a uniformly covered decomposition scaffold, uses a spatiotemporal encoder to represent physical and task-dependent interactions, and combines temporal and structural relevance to construct agent-specific advantages. Experiments on SMAC, GRF, Traffic Junction, and MPE demonstrate consistent improvements over MAPPO-style, value-factorization, and dynamic-grouping baselines. On the hardest SMAC settings, HySTAR achieves relative gains of 16.7\% over MAPPO and 15.6\% over HYGMA, ranks first on all six GRF scenarios, reduces Traffic Junction convergence epochs by up to 40.2\% relative to MAGIC, and obtains the highest MPE episode rewards. Controlled topology, agent-death, neighborhood, and parameter analyses support the benefit of anchoring the decomposition scaffold while adapting the propagated representations.

Problem

• Cooperative Multi-Agent Reinforcement Learning
• Credit Assignment
• Structural Target Drift
• Partial Observability
• Value Decomposition

Innovation

• Cooperative Multi-Agent Reinforcement Learning
• Hypergraph
• Credit Assignment
• Value Decomposition
• Structural Target Drift