FedRLHF: A Convergence-Guaranteed Federated Framework for Privacy-Preserving and Personalized RLHF

2024-12-20

arXiv.org

AI Summary

To address privacy leakage and insufficient personalization arising from centralized RLHF—which requires aggregating raw data and human feedback—this paper proposes Federated Reinforcement Learning with Human Feedback (FedRLHF). FedRLHF enables collaborative policy learning across multiple clients without sharing raw data or feedback: each client trains a personalized reward model using local human feedback and optimizes its policy accordingly, while distributed policy updates are aggregated via a convergence-driven mechanism. Theoretically, we establish the first convergence guarantee for FedRLHF and derive a scalable sample complexity bound. Empirically, on MovieLens and IMDb datasets, FedRLHF matches the performance of centralized RLHF while significantly improving cross-client personalization and strictly preserving data privacy.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Humans and AI: Human-in-the-loop Machine Learning
• Multiagent Systems: Adversarial Agents

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

In the era of increasing privacy concerns and demand for personalized experiences, traditional Reinforcement Learning with Human Feedback (RLHF) frameworks face significant challenges due to their reliance on centralized data. We introduce Federated Reinforcement Learning with Human Feedback (FedRLHF), a novel framework that decentralizes the RLHF process. FedRLHF enables collaborative policy learning across multiple clients without necessitating the sharing of raw data or human feedback, thereby ensuring robust privacy preservation. Leveraging federated reinforcement learning, each client integrates human feedback locally into their reward functions and updates their policies through personalized RLHF processes. We establish rigorous theoretical foundations for FedRLHF, providing convergence guarantees, and deriving sample complexity bounds that scale efficiently with the number of clients. Empirical evaluations on the MovieLens and IMDb datasets demonstrate that FedRLHF not only preserves user privacy but also achieves performance on par with centralized RLHF, while enhancing personalization across diverse client environments.

Problem

• Decentralizes RLHF for privacy preservation
• Ensures personalized RLHF across multiple clients
• Provides convergence guarantees for federated learning

Innovation

• Decentralized RLHF framework
• Privacy-preserving federated learning
• Personalized policy integration