Federated Sketching LoRA: On-Device Collaborative Fine-Tuning of Large Language Models

2025-01-31

AI Summary

In collaborative fine-tuning of large language models (LLMs) across resource-heterogeneous edge devices, disparities in computational capacity hinder effective LoRA rank adaptation, constrain parameter updates, and degrade training efficiency. Method: We propose a dynamic submatrix update framework based on randomized matrix sketching, enabling edge devices to selectively update local LoRA submodules on-demand. Our approach unifies LoRA, federated learning, and convergence analysis to jointly optimize communication, computation, and model performance. Contribution/Results: We establish, for the first time, a theoretical relationship between the sketching ratio and convergence rate. Extensive experiments across multiple LLMs and datasets demonstrate that our method significantly outperforms existing federated fine-tuning approaches—reducing communication overhead by 32%–57%, accelerating training by 1.8×–3.4×, and preserving model accuracy.

Technology Category

• Machine Learning: Learning on the Edge & Model Compression
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization

Abstract

Fine-tuning large language models (LLMs) on devices is attracting increasing interest. Recent works have fused low-rank adaptation (LoRA) techniques with federated fine-tuning to mitigate challenges associated with device model sizes and data scarcity. Still, the heterogeneity of computational resources remains a critical bottleneck: while higher-rank modules generally enhance performance, varying device capabilities constrain LoRA's feasible rank range. Existing approaches attempting to resolve this issue either lack analytical justification or impose additional computational overhead, leaving a wide gap for an efficient and theoretically-grounded solution. To address these challenges, we propose federated sketching LoRA (FSLoRA), which leverages a sketching mechanism to enable devices to selectively update submatrices of global LoRA modules maintained by the server. By adjusting the sketching ratios, which determine the ranks of the submatrices on the devices, FSLoRA flexibly adapts to device-specific communication and computational constraints. We provide a rigorous convergence analysis of FSLoRA that characterizes how the sketching ratios affect the convergence rate. Through comprehensive experiments on multiple datasets and LLM models, we demonstrate FSLoRA's superior performance compared to various baselines.

Problem

• Multi-device Collaboration
• Large Language Model Adjustment
• Performance Disparity

Innovation

• FSLoRA
• Adaptive Federated Learning
• LoRA Integration