FedGSA: Geometry-Consistent Subspace Aggregation for Differentially Private Federated LoRA

2026-08-04

AI Summary

This work addresses the distortion and noise accumulation in differentially private federated LoRA caused by basis-dependent client updates and Euclidean-space aggregation. To overcome these limitations, the authors propose FedGSA, a novel framework that models private LoRA updates as basis-invariant subspaces on the Grassmann manifold. By encoding dominant update directions via projection matrices and performing geometrically consistent subspace aggregation at the server, FedGSA eliminates distortions arising from basis misalignment, privacy-preserving noise, and client heterogeneity—without incurring additional privacy loss. Empirical evaluations demonstrate that FedGSA consistently outperforms existing methods across four GLUE tasks and language generation benchmarks, achieving average accuracy gains of 2.17% and 2.27% under privacy budgets ε=6 and ε=3, respectively.

Abstract

Low-Rank Adaptation (LoRA) enables communication-efficient federated fine-tuning of pretrained language models. However, integrating differential privacy (DP) into federated LoRA remains challenging: independently perturbing and aggregating its two low-rank matrices can cause aggregation mismatch and the quadratic noise term. Existing methods mitigate these issues by freezing one low-rank matrix but still rely on Euclidean aggregation, which is basis-dependent and may distort the global update. To address this limitation, we propose FedGSA, a geometry-consistent aggregation framework for differentially private federated LoRA. FedGSA represents each privatized client update as a basis-invariant subspace on the Grassmann manifold. In each communication round, clients extract low-dimensional subspaces capturing dominant update directions and encode them as projection matrices. The server aggregates these representations to estimate a geometry-consistent global update subspace and reconstructs the global LoRA factors within it, reducing distortion caused by basis misalignment, privacy noise, and heterogeneous client updates. We prove that FedGSA incurs no additional privacy loss beyond client-side DP training and establish its convergence under standard assumptions. Experiments on four GLUE tasks and a language generation benchmark demonstrate consistent improvements across privacy budgets and degrees of data heterogeneity. In particular, FedGSA improves average accuracy over the strongest baseline by 2.17% and 2.27% under $ε=6$ and $ε=3$, respectively.

Problem

• Federated Learning
• Low-Rank Adaptation
• Differential Privacy
• Geometry-Consistent Aggregation
• Grassmann Manifold

Innovation

• Geometry-Consistent Aggregation
• Grassmann Manifold
• Differentially Private Federated Learning
• LoRA
• Subspace Representation