FedLAFP: Low-Rank Aggregation Meets Full-Rank Personalization in Federated Fine-Tuning

2026-09-29

AI Summary

This study addresses the challenge that statistical heterogeneity in federated fine-tuning renders a single global low-rank adapter insufficient for personalized demands. To this end, we propose a role-aware framework that decouples the adaptation architecture into a globally aggregated LoRA branch and a private RandLoRA branch, achieving efficient personalization through fusion with mixing coefficients. The core innovation lies in leveraging random bases to enhance the representational capacity of the private branch while exclusively exchanging shared parameters, thereby incurring zero additional communication overhead. Experimental results demonstrate that the proposed method attains an average personalized accuracy of 86.93% across four vision benchmarks, outperforming the strongest baseline by 1.3 percentage points. Consequently, this work achieves a superior trade-off between communication efficiency and model performance in federated personalized learning scenarios.

Abstract

Federated parameter-efficient fine-tuning enables clients to adapt pre-trained models without sharing raw data or communicating the full model, but statistical heterogeneity makes a single global adapter insufficient for personalized prediction. Existing personalized methods typically use the same low-rank structure for both shared and private adaptation, overlooking their distinct requirements for aggregation and personalization. We propose FedLAFP, a role-aware framework that couples a compact, globally aggregated LoRA branch with a client-private, full-rank-capable RandLoRA branch. The shared branch provides an efficient interface for transferring common knowledge, whereas the private branch combines fixed random low-rank bases with learned scaling coefficients to provide expressive client-specific adaptation without additional communication. Client- and layer-specific mixing coefficients jointly fuse the two branches, and only the shared LoRA parameters are exchanged. A controlled linear study supports this role assignment: LoRA yields more aligned client updates and lower aggregation error, while RandLoRA more accurately recovers client-specific residuals. Experiments across four visual recognition benchmarks show that FedLAFP consistently outperforms local-only and federated LoRA baselines, achieving an average personalized accuracy of $86.93\%$ and exceeding the best baseline average by $1.30$ percentage points.

Problem

• Federated Learning
• Parameter-Efficient Fine-Tuning
• Statistical Heterogeneity
• Personalization
• Low-Rank Adaptation

Innovation

• Federated Fine-Tuning
• Low-Rank Adaptation (LoRA)
• Personalization
• RandLoRA
• Parameter-Efficient