Adaptive Parameter-Efficient Federated Fine-Tuning on Heterogeneous Devices

2024-12-28

AI Summary

To address the challenges of resource-constrained and device-heterogeneous environments in Federated Fine-Tuning (FedFT), this paper proposes LEGEND, an adaptive LoRA configuration framework. LEGEND is the first to uncover the coupling between LoRA depth (i.e., number of tuned layers) and rank distribution across layers, and introduces a dynamic configuration algorithm tailored for heterogeneous devices that jointly optimizes both LoRA depth and per-layer rank allocation. By integrating parameter-efficient fine-tuning, federated learning, and heterogeneous system modeling, LEGEND achieves end-to-end deployment on 80 commercial edge devices. Compared to state-of-the-art methods, it accelerates training by 1.5–2.8× and reduces communication overhead by 42.3%, while preserving model accuracy. LEGEND thus demonstrates significant advantages in computational efficiency, adaptability to hardware diversity, and practical deployability.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Search and Optimization: Learning to Search
• Computer Vision: Large Vision Models

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Federated Web and WoT systems, including distributed, federated and edge-based data processing
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Federated fine-tuning (FedFT) has been proposed to fine-tune the pre-trained language models in a distributed manner. However, there are two critical challenges for efficient FedFT in practical applications, i.e., resource constraints and system heterogeneity. Existing works rely on parameter-efficient fine-tuning methods, e.g., low-rank adaptation (LoRA), but with major limitations. Herein, based on the inherent characteristics of FedFT, we observe that LoRA layers with higher ranks added close to the output help to save resource consumption while achieving comparable fine-tuning performance. Then we propose a novel LoRA-based FedFT framework, termed LEGEND, which faces the difficulty of determining the number of LoRA layers (called, LoRA depth) and the rank of each LoRA layer (called, rank distribution). We analyze the coupled relationship between LoRA depth and rank distribution, and design an efficient LoRA configuration algorithm for heterogeneous devices, thereby promoting fine-tuning efficiency. Extensive experiments are conducted on a physical platform with 80 commercial devices. The results show that LEGEND can achieve a speedup of 1.5-2.8$ imes$ and save communication costs by about 42.3% when achieving the target accuracy, compared to the advanced solutions.

Problem

• Federated Learning
• Resource-constrained Environment
• Model Fine-tuning

Innovation

• LEGEND Framework
• LoRA Optimization
• Efficient Federated Learning