FGRPO: Federated GRPO with Adaptive Aggregation on Non-IID Data

2026-06-01

AI Summary

This work addresses the instability of distributed reinforcement learning under non-independent and identically distributed (Non-IID) data by proposing Federated GRPO (FGRPO), a framework for decentralized fine-tuning of inference language models that preserves local data privacy. FGRPO integrates the actor-only GRPO algorithm with a relative performance gain evaluation mechanism based on personalized historical baselines to dynamically decouple the difficulty of heterogeneous tasks. Leveraging this assessment, the framework employs an adaptive model aggregation strategy tailored to task heterogeneity. Experimental results demonstrate that FGRPO significantly enhances training stability, convergence efficiency, and final performance in Non-IID settings.

Abstract

Recent advances in language models have established reinforcement learning as the primary paradigm for eliciting self-correction and long-chain reasoning. While group relative policy optimization (GRPO) offers superior scalability by eliminating the critic network, deploying it on a central infrastructure entails collecting a large volume of data from distributed owners, which poses significant privacy risks. To address these concerns, we introduce federated GRPO (FGRPO), a framework designed to decentralize the fine-tuning of reasoning models across heterogeneous data owners. To effectively mitigate the instability caused by divergent reward scales across heterogeneous tasks, FGRPO incorporates an adaptive aggregation mechanism based on relative performance gain. By characterizing each client's improvement relative to its personalized historical baseline, the framework dynamically prioritizes effective learning trajectories regardless of local task difficulty. FGRPO ensures robust convergence on non-IID data while preserving data privacy.

Problem

• Federated Learning
• Non-IID Data
• Privacy Preservation
• Reinforcement Learning
• Reward Heterogeneity

Innovation

• Federated Learning
• GRPO
• Adaptive Aggregation
• Non-IID Data
• Privacy-Preserving Reinforcement Learning