Revisiting Privacy Amplification by Subsampling in Selective Release DPSGD

2026-06-02

AI Summary

This work addresses the low utility and slow convergence of existing differentially private stochastic gradient descent (DPSGD) methods, as well as the inadequate privacy analysis in selective release mechanisms like DPSUR, which neglects the impact of gradient clipping on sampling probabilities and thus yields non-rigorous privacy guarantees. The paper proposes DPSR-CG, an algorithm that, for the first time, provides a rigorous privacy analysis accounting for the dynamic sampling behavior induced by gradient clipping, thereby closing the privacy gap in prior approaches. By integrating an improved privacy amplification theory, DPSR-CG enables precise computation of the privacy budget. Empirical evaluations across multiple benchmarks—including MNIST, CIFAR-10, FMNIST, and IMDB—demonstrate that DPSR-CG significantly enhances model utility and convergence speed while strictly preserving differential privacy.

Abstract

Machine learning's reliance on sensitive data necessitates privacy-preserving techniques like Differentially Private Stochastic Gradient Descent (DPSGD). However, DPSGD suffers from substantial utility degradation and slow convergence due to gradient clipping and noise injection. Prior works have attempted to improve DPSGD from various perspectives; notably, the Differentially Private Selective Update and Release (DPSUR) algorithm has achieved remarkable model utility. However, the privacy accounting in DPSUR overlooks the variation in sampling probability introduced by the selective release mechanism, which compromises the rigor of its privacy guarantees. To address these limitations, we re-evaluate the privacy analysis of the selective release mechanism and propose a novel algorithm: Differentially Private Selective Release based on Clipped Gradients (DPSR-CG). Through a rigorous, newly derived privacy analysis and extensive experiments on multiple datasets (MNIST, CIFAR-10, IMDB, and FMNIST), we demonstrate that our DPSR-CG mechanism maintains strict privacy guarantees while achieving exceptional model performance.

Problem

• Differential Privacy
• Privacy Amplification
• Selective Release
• DPSGD
• Privacy Accounting

Innovation

• privacy amplification
• selective release
• differential privacy
• subsampled SGD
• clipped gradients