Technology Category
Machine Learning: Distributed Machine Learning & Federated LearningMultiagent Systems: Adversarial AgentsSearch and Optimization: Learning to Search
The Actor-Critic Update Order Matters for PPO in Federated Reinforcement Learning
🤖 AI Summary
In federated reinforcement learning (FRL), Proximal Policy Optimization (PPO) suffers from convergence degradation due to misalignment between local actor-critic update order and global model aggregation—particularly under data heterogeneity, where divergent critic estimates induce conflicting gradient directions. To address this, we propose FedRAC, the first FRL framework that reverses PPO’s standard update order to “actor before critic.” This design theoretically eliminates critic estimation bias across heterogeneous clients, yielding a convergence bound independent of data heterogeneity. FedRAC integrates policy-gradient-based actor updates, delayed critic synchronization, and rigorous convergence analysis. We evaluate it on three standard RL benchmarks and a highly heterogeneous SUMO autonomous driving task. Results show average cumulative reward improvements of 12.7%–23.4%, 1.8×–2.5× faster convergence, and significantly enhanced robustness and practicality.
Technology Category
Application Category
📝 Abstract
In the context of Federated Reinforcement Learning (FRL), applying Proximal Policy Optimization (PPO) faces challenges related to the update order of its actor and critic due to the aggregation step occurring between successive iterations. In particular, when local actors are updated based on local critic estimations, the algorithm becomes vulnerable to data heterogeneity. As a result, the conventional update order in PPO (critic first, then actor) may cause heterogeneous gradient directions among clients, hindering convergence to a globally optimal policy. To address this issue, we propose FedRAC, which reverses the update order (actor first, then critic) to eliminate the divergence of critics from different clients. Theoretical analysis shows that the convergence bound of FedRAC is immune to data heterogeneity under mild conditions, i.e., bounded level of heterogeneity and accurate policy evaluation. Empirical results indicate that the proposed algorithm obtains higher cumulative rewards and converges more rapidly in five experiments, including three classical RL environments and a highly heterogeneous autonomous driving scenario using the SUMO traffic simulator.
Problem
Research questions and friction points this paper is trying to address.
PPO actor-critic update order challenges in Federated Reinforcement Learning
Data heterogeneity causes divergent gradients in conventional PPO updates
Reversed update order needed for global policy convergence in FRL
Innovation
Methods, ideas, or system contributions that make the work stand out.
Reverses actor-critic update order in PPO
Proposes FedRAC for Federated Reinforcement Learning
Ensures convergence despite data heterogeneity
🔎 Similar Papers
No similar papers found.
