Recent Advances in Federated Learning Driven Large Language Models: A Survey on Architecture, Performance, and Security

2024-06-14

AI Summary

Federated learning (FL) of large language models (LLMs) faces critical challenges in privacy preservation, communication overhead, and security—particularly concerning the “right to be forgotten” and machine unlearning (MU). Method: We systematically establish the first comprehensive taxonomy of MU for federated LLMs, categorizing approaches into perturbation-based, model-decomposition, and incremental retraining strategies, and analyze their trade-offs among utility, privacy, and efficiency. Integrating differential privacy, low-rank decomposition, and incremental fine-tuning, we propose a deployable, secure evolutionary framework for federated LLMs. Contribution/Results: Through multi-scenario empirical evaluation, we quantitatively assess communication cost, unlearning completeness, and post-unlearning model utility across MU methods. Our work provides both theoretical foundations and practical guidelines for privacy-compliant, scalable federated training of LLMs.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Natural Language Processing: (Large) Language Models
• Humans and AI: Human-in-the-loop Machine Learning

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Security and Privacy: Security and privacy of machine learning and AI applications

Abstract

Federated Learning (FL) offers a promising paradigm for training Large Language Models (LLMs) in a decentralized manner while preserving data privacy and minimizing communication overhead. This survey examines recent advancements in FL-driven LLMs, with a particular emphasis on architectural designs, performance optimization, and security concerns, including the emerging area of machine unlearning. In this context, machine unlearning refers to the systematic removal of specific data contributions from trained models to comply with privacy regulations such as the Right to be Forgotten. We review a range of strategies enabling unlearning in federated LLMs, including perturbation-based methods, model decomposition, and incremental retraining, while evaluating their trade-offs in terms of efficiency, privacy guarantees, and model utility. Through selected case studies and empirical evaluations, we analyze how these methods perform in practical FL scenarios. This survey identifies critical research directions toward developing secure, adaptable, and high-performing federated LLM systems for real-world deployment.

Problem

• Surveying FL-driven LLMs' architecture, performance, and security challenges
• Exploring machine unlearning for privacy compliance in federated LLMs
• Evaluating unlearning methods' trade-offs in efficiency, privacy, and utility

Innovation

• Federated Learning for decentralized LLM training
• Machine unlearning for privacy compliance
• Perturbation-based and decomposition unlearning methods