Noise-Aware Shrinkage for Differentially Private Zeroth-Order Fine-Tuning of Large Language Models

2026-08-04

AI Summary

This work addresses a critical limitation in existing differentially private zeroth-order optimization methods, which often neglect variations in signal strength during gradient aggregation, leading to noise-dominated updates that degrade model utility. To mitigate this issue, the authors propose SAGE, a novel approach that introduces, for the first time, a noise-aware adaptive shrinkage mechanism. SAGE dynamically attenuates privatized updates by online estimation of signal energy and adaptive adjustment of a bounded shrinkage factor. Notably, this method achieves significant performance gains without consuming additional privacy budget or increasing query complexity. Empirical evaluations on RoBERTa-large, OPT-1.3B, and OPT-6.7B demonstrate that SAGE consistently outperforms current baselines under identical privacy guarantees, while maintaining memory efficiency during forward propagation.

Abstract

Differentially private zeroth-order optimization (DP-ZO) enables memory-efficient private fine-tuning of large language models using only forward evaluations. Existing aggregation-based DP-ZO methods reconstruct model updates at a fixed scale, ignoring that the strength of useful signals varies throughout training. Consequently, noise-dominated updates may receive excessive weight and degrade model utility. To address this issue, we propose SAGE, a noise-aware shrinkage method that adaptively attenuates privatized estimates according to their estimated signal quality. SAGE subtracts the known Gaussian noise variance from the observed second moment to estimate the underlying signal energy, stabilizes this estimate through temporal tracking, and compares its current signal-to-noise level with a warm-up reference to derive a bounded shrinkage factor. As pure post-processing, SAGE requires neither additional privacy budget nor model queries and introduces only constant additional state. Our theoretical analysis shows that shrinkage reduces the quadratic update-risk term faster than the linear descent term, preserving useful descent while limiting the influence of noise-dominated updates. Experiments on RoBERTa-large, OPT-1.3B, and OPT-6.7B demonstrate that SAGE outperforms existing baselines in most settings under the same privacy budgets while preserving the forward-only memory efficiency of DP-ZO.

Problem

• differentially private optimization
• zeroth-order optimization
• large language models
• noise-dominated updates
• model utility

Innovation

• noise-aware shrinkage
• differentially private optimization
• zeroth-order fine-tuning
• signal-to-noise estimation
• adaptive update attenuation