🤖 AI Summary
This work addresses the significant performance degradation commonly observed in real-world multi-agent reinforcement learning (MARL) systems due to observation and communication delays as well as packet loss. To mitigate these issues, the authors propose a modular, plug-and-play state estimation layer that replaces delayed observations with belief states during execution. This layer integrates a learned gated dynamic model with a recursive Kalman filter to enable robust estimation of instantaneous states under asynchronous and incomplete measurements. Notably, the approach requires no modifications to the underlying MARL algorithm, architecture, or reward function. Experimental results demonstrate that the method substantially enhances policy robustness against communication delays and packet loss across multiple multi-agent continuous control benchmarks, with particularly strong performance in scenarios requiring tight coordination or exhibiting dynamic instability.
📝 Abstract
Real-world multi-agent reinforcement learning (MARL) systems must often operate under stale observations, stochastic communication delays, and intermittent packet loss. Policies trained under idealized synchronous conditions frequently exhibit significant performance degradation in these regimes because they act on outdated feedback. We propose a modular execution-stage state-estimation layer that replaces delayed communicated observations with current belief-state estimates. The framework integrates a learned Gated transition model with a recursive Kalman filtering layer to estimate instantaneous states from asynchronous measurements. A primary advantage of this approach is its modularity, The estimator serves as a plug-in for pre-trained policies, requiring no modifications to the original MARL training algorithm, architecture, or reward structure. Evaluation across diverse multi-agent and continuous-control benchmarks demonstrates that the proposed layer consistently enhances robustness to communication latency and message loss. The most significant performance gains are observed in coordination-intensive and dynamically unstable tasks where temporal consistency is critical for control.