PRISM: Gauge-Invariant Tangent-Space Differentially Private LoRA

2026-05-30

AI Summary

Direct application of DP-SGD to LoRA suffers from norm-dependent noise amplification due to parameterization non-uniqueness, which disrupts the privacy-utility trade-off. To address this, this work proposes PRISM, a mechanism that constructs norm-invariant differentially private perturbations within the tangent space of LoRA, thereby avoiding bilinear noise amplification and enabling efficient low-dimensional sampling. PRISM is the first method to achieve intrinsic norm-invariant differential privacy in LoRA, offering a closed-form characterization of effective noise and incorporating a privacy-aware adaptive update rule. Experimental results demonstrate that PRISM significantly improves fine-tuning utility and numerical stability while maintaining rigorous $(\varepsilon, \delta)$-differential privacy guarantees.

Abstract

Applying differential privacy (DP) via DP-SGD to Low-Rank Adaptation (LoRA) is a natural approach for privacy-preserving fine-tuning. However, LoRA's low-rank parameterization poses a fundamental challenge. In LoRA, each trainable update is represented as a low-rank matrix $Z = AB^\top$, but this factorization is inherently non-identifiable: many factor pairs $(A,B)$ represent the same update $Z$. As a result, applying DP-SGD directly to the factors induces gauge-dependent perturbations on $Z$, and we show that this naive DP-LoRA can lead to unbounded noise amplification. We propose PRISM, an intrinsic DP mechanism for LoRA that is gauge invariant by construction, avoids bilinear noise amplification, and admits an efficient low-dimensional noise sampler. Moreover, PRISM yields a closed-form characterization of the effective intrinsic noise induced on $Z$, enabling stable privacy-utility trade-offs through bounded, gauge-invariant perturbations. We establish standard $(ε,δ)$-DP guarantees for PRISM and introduce a DP-aware, gauge-invariant adaptive update rule that prevents adaptive optimization from amplifying injected privacy noise, improving numerical stability in practice.

Problem

• differential privacy
• LoRA
• gauge invariance
• noise amplification
• low-rank adaptation

Innovation

• gauge invariance
• differential privacy
• LoRA
• tangent space
• noise amplification