FraQ: Efficient Coordinate-Space Recompression for Federated Low-Rank Adaptation

2026-08-04

AI Summary

This work addresses the spatial mismatch in client updates during federated LoRA training caused by dual-factor parameterization, which degrades model performance. To resolve this, the authors propose an efficient server-side coordinate space recompression method that reconstructs the global low-rank adapter accurately without explicit dense matrix decomposition. By leveraging orthogonal basis decomposition and coordinate matrix reconstruction—combined with singular spectrum estimation of the Gram matrix and an energy-threshold-based adaptive rank selection—the approach achieves precise aggregation while maintaining communication efficiency. This is the first method in federated LoRA to simultaneously preserve aggregation accuracy and reduce communication overhead. Empirical results show it nearly matches the performance of uncompressed baselines on text classification and commonsense reasoning tasks, while substantially lowering downlink communication costs and server-side computational demands.

Abstract

Federated fine-tuning with Low-Rank Adaptation (LoRA) enables efficient collaborative adaptation of Large Language Models (LLMs) without centralizing private data. However, LoRA's two-factor parameterization creates an aggregation mismatch across clients: naively averaging the factors does not recover the average of their induced updates. This mismatch can be avoided by forming the exact aggregate in the full weight space and then recompressing it, but decomposing the resulting dense matrix is computationally expensive and memory-intensive. We propose FraQ, an efficient coordinate-space recompression method for federated LoRA. Starting from stacked factors that exactly represent the aggregate, FraQ factorizes it into an orthonormal basis and a compact coordinate matrix. It then recovers the singular spectrum from a small Gram matrix, selects the smallest rank satisfying a prescribed energy threshold, and maps the selected coordinate subspace back through the basis to construct the global adapter. Experiments on text classification and commonsense reasoning benchmarks show that FraQ achieves accuracy close to uncompressed baselines while substantially reducing downlink communication with low server-side recompression overhead.

Problem

• Federated Learning
• Low-Rank Adaptation
• Parameter Aggregation
• Model Compression
• Communication Efficiency

Innovation

• Federated Learning
• Low-Rank Adaptation
• Coordinate-Space Recompression
• Communication Efficiency
• LoRA Aggregation