Fully Byzantine-Resilient Multi-Agent Reinforcement Learning

2026-09-22

AI Summary

研究分布式拜占庭容错多智能体强化学习，提出FRAC-MARL方法利用两跳消息识别可靠信息，确保参数在攻击下仍收敛到无攻击状态。

Abstract

We study distributed Byzantine-resilient actor-critic multi-agent reinforcement learning (AC-MARL), where agents collectively learn policies through local interactions. Existing methods guarantee convergence of the agents' parameters only to a neighborhood of the attack-free limit points, resulting in degraded performance. We propose Fully Resilient AC-MARL (FRAC-MARL), a decentralized method in which each agent leverages redundancy in two-hop messages to identify reliable messages. Under linear parameterizations of the value and team-reward functions and Byzantine edge attacks, where adversarial behavior is confined to the communication layer, we prove that agents' parameters converge almost surely to the same limit points as in the attack-free case over time-varying communication graphs. We introduce a novel topological condition for the convergence of our method, present a systematic method to construct such networks, and prove that this condition can be verified in polynomial time. Finally, we demonstrate our method on cooperative multi-robot formation control tasks.

Problem

• Byzantine-resilient
• multi-agent reinforcement learning
• convergence

Innovation

• Fully Resilient AC-MARL
• Byzantine-resilient
• two-hop messages
• reliable message identification
• topological condition