EcoLoRA: Communication-Efficient Federated Fine-Tuning of Large Language Models

2025-05-20

Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing

AI Summary

To address the high communication overhead bottleneck in fine-tuning large language models (LLMs) within federated learning (FL), this paper proposes a lightweight, efficient, and lossless distributed fine-tuning framework. Methodologically, it introduces (1) a novel rotating LoRA segmentation sharing mechanism that dynamically schedules uploads of low-rank adaptation modules across training rounds, and (2) a synergistic communication–computation optimization combining adaptive gradient sparsification with entropy coding–based lossless compression. Empirically, the framework achieves state-of-the-art (SOTA) performance while reducing communication time by up to 79% and total training time by 65%. Its effectiveness and robustness are validated across diverse downstream tasks, including question answering and value alignment.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

To address data locality and privacy restrictions, Federated Learning (FL) has recently been adopted to fine-tune large language models (LLMs), enabling improved performance on various downstream tasks without requiring aggregated data. However, the repeated exchange of model updates in FL can result in prohibitively high communication costs, hindering the distributed learning process. To address this challenge, we propose EcoLoRA, a novel communication-efficient federated fine-tuning framework for LLMs. Leveraging the modular structure, we propose a round-robin segment sharing scheme, where each client uploads only a complementary LoRA segment per round to reduce network bandwidth. It is further combined with adaptive sparsification methods tailored to LoRA's training dynamics and lossless encoding techniques. We conduct extensive evaluations on both question-answering and value-alignment tasks across multiple datasets and models. The results show that EcoLoRA significantly reduces communication overhead without compromising performance. For instance, it reduces communication time by up to 79% and total training time by up to 65%.

Problem

• Reducing communication costs in federated fine-tuning of large language models
• Optimizing bandwidth usage through segment sharing and sparsification techniques
• Maintaining model performance while minimizing distributed training overhead

Innovation

• Round-robin segment sharing reduces client uploads
• Adaptive sparsification tailored to LoRA training dynamics
• Combines lossless encoding with modular parameter transmission