🤖 AI Summary
Heterogeneous client data in federated LoRA leads to misalignment of low-rank update subspaces, causing aggregation conflicts and slow convergence. This work is the first to reveal the critical impact of such subspace misalignment on federated LoRA performance and proposes a subspace regularization method that explicitly aligns clients’ geometric structures by constraining local updates to remain close to a shared global reference subspace. Leveraging SVD-based basis alignment analysis, the proposed approach significantly outperforms baseline methods such as FedAvg on RoBERTa-large, achieving an accuracy of 0.429 ± 0.011 and a subspace basis overlap of 0.9999, thereby demonstrating the effectiveness of high-precision subspace alignment.
📝 Abstract
Federated low-rank adaptation methods are attractive for fine-tuning large models under communication and privacy constraints, but heterogeneous client data can induce geometric misalignment between local low-rank updates. We study whether this subspace misalignment leads to destructive aggregation and slower convergence in LoRA-based federated learning. We propose a subspace-regularized federated LoRA objective that encourages local client updates to remain close to a shared global reference subspace. We present a complete empirical evaluation on two pretrained models, RoBERTa-large and SmolLM-360M, over HellaSwag in a non-IID 10-client federated setting, across 3 random seeds (42, 43, 44), yielding 24 total experimental runs (4 methods x 3 seeds x 2 models). On RoBERTa-large, Subspace-Reg achieves the strongest mean best accuracy (0.454 +/- 0.023), mean final accuracy (0.429 +/- 0.011), and lowest final loss (1.363) across all three seeds, outperforming FedAvg, SVD redistribution, and FedSVD baselines by a large margin. On SmolLM-360M, FedAvg leads on accuracy, revealing that accuracy gains are model-dependent. Crucially, Subspace-Reg achieves near-perfect basis overlap, approximately 0.9999, on both models and across all seeds, versus 0.958 to 0.991 for all baselines, providing robust support for the geometric alignment hypothesis. The code is publicly available at https://github.com/sadia-sigma-lab/Subspace-Constrained-Federated-learning-with-Lora.