Aggregate in the Advantage, Not the Ratio: A Canonical-Form Analysis of Cooperative Multi-Agent Policy Optimization

2026-07-20

AI Summary

This work addresses a fundamental challenge in multi-agent policy optimization: the selection of neighboring agents for aggregating advantage functions and importance weight ratios to enable efficient collaboration. The paper introduces a support matrix formalism for aggregation and demonstrates that the optimization objective depends solely on the product of the advantage and ratio matrices, thereby revealing the intrinsic structural properties of aggregation design. Through bias-variance analysis and theoretical derivation, the study shows that aggregating advantage functions with coupled neighborhood information while preserving individualized importance weight ratios prevents exponential variance growth. This insight leads to the principled design guideline of “aggregate advantages, individualize ratios,” offering both theoretical grounding and practical guidance for developing effective multi-agent reinforcement learning algorithms.

Abstract

Multi-agent policy optimization, exemplified by PPO-based methods, is a key branch of cooperative Multi-Agent Reinforcement Learning (MARL). A central design question is how many neighboring agents\footnote{In this paper, "neighbors" refer not only to physical proximity but also to agents whose actions influence one another.} to aggregate in order to effectively utilize global information for cooperation. This decision must be made along two dimensions: in the advantage (which agents' rewards contribute to the credit signal) and in the ratio (which agents' likelihood ratios form the clipped importance weight). Existing methods occupy scattered, underexplored points on these two axes: IPPO treats both separately; MAPPO pairs a team-level advantage with per-agent ratios; HAPPO employs sequential ratios with per-agent advantages; and single-agent reductions operating on factorized joint policies aggregate both into fully joint products. We formalize these two design choices as support matrices $\SA$ and $\SR$, and prove a canonical structure: the expected multi-agent policy optimization objective depends on the pair $(\SA,\SR)$ only through their matrix product $\tS=\SR\SA$. This yields two key consequences: (i) Redundancy: the two support matrices are interchangeable with respect to the signal, meaning neither aggregation pattern is inherently superior.(ii) Variance Ordering: the advantage aggregates rewards as a sum (additive variance with an interior bias-variance optimum at the coupling neighborhood), whereas the ratio aggregates likelihood ratios as a product (multiplicative variance that grows exponentially with support size, with no accompanying bias reduction). The resulting design principle is unambiguous: aggregate neighbors in the advantage, sized to the coupling neighborhood, and keep the ratio per-agent.

Problem

• multi-agent reinforcement learning
• policy optimization
• advantage aggregation
• likelihood ratio
• cooperative MARL

Innovation

• multi-agent reinforcement learning
• policy optimization
• advantage aggregation
• likelihood ratio
• canonical form