Three-Pronged Spectral Control for Federated Parameter Efficient Fine Tuning

2026-07-23

AI Summary

This work addresses the challenge of high aggregation variance and poor transfer performance in federated parameter-efficient fine-tuning under non-IID data, caused by spectral misalignment in updated subspaces. To mitigate this, the authors propose TRISHUL, a novel framework that introduces, for the first time, a spectral control triad: algebraically precise aggregation via shared frozen multi-head low-rank bases, nuclear-norm proximal shrinkage to suppress high-rank components before upload, and non-uniform adapter head allocation across layers using a concave water-filling rule informed by pre-trained layer capacities. Without increasing communication overhead, TRISHUL significantly enhances robustness, consistently outperforming federated LoRA across benchmarks including CIFAR-100, SVHN, 20 Newsgroups, MRQA, and GLUE—particularly excelling in highly heterogeneous settings with faster convergence, greater stability, and superior final accuracy.

Abstract

Federated parameter-efficient fine-tuning (PEFT) enables communication-efficient adaptation of large pretrained models on decentralized edge data, but it remains fragile under non-IID client heterogeneity. In low-rank adaptation (LoRA), different clients may learn locally useful but spectrally misaligned update subspaces, causing high-variance aggregation and poor global transfer. We propose TRISHUL, a spectral-control framework for robust federated PEFT. TRISHUL follows the FL no-raw-data-sharing setting but does not itself provide formal privacy guarantees. TRISHUL uses shared frozen multi-head low-rank bases to obtain algebraically exact aggregation of compact core updates, applies nuclear norm proximal shrinkage to suppress client-specific high-rank spectral components before upload, and allocates adaptation heads non-uniformly across layers using a concave water filling budget rule derived from pretrained layer capacity. Because shrinkage is performed only on small core matrices, TRISHUL adds negligible computation and no extra per-round communication over the underlying multi-head PEFT protocol. Across vision and language benchmarks, including CIFAR-100, SVHN, 20 Newsgroups, MRQA, and GLUE with LLaMA3.2-1B, TRISHUL improves convergence, stability, and final performance over federated LoRA baselines, with greater gains under stronger heterogeneity.

Problem

• federated learning
• parameter-efficient fine-tuning
• non-IID heterogeneity
• spectral misalignment
• low-rank adaptation

Innovation

• Federated Learning
• Parameter-Efficient Fine-Tuning
• Spectral Control
• Low-Rank Adaptation
• Non-IID Heterogeneity