AdaDPIGU: Differentially Private SGD with Adaptive Clipping and Importance-Based Gradient Updates for Deep Neural Networks

2025-07-08

AI Summary

Existing differentially private stochastic gradient descent (DP-SGD) suffers significant performance degradation in high-dimensional settings due to noise scaling linearly with dimensionality. This work proposes AdaDPIGU, a privacy-preserving training framework for deep neural networks, which—uniquely—integrates parameter importance estimation with coordinate-wise adaptive clipping. Specifically, it leverages the differentially private Gaussian mechanism to estimate the importance of each model parameter coordinate, enabling sparse gradient updates and pruning of low-importance coordinates. The method rigorously satisfies $(varepsilon,delta)$-differential privacy while substantially mitigating noise accumulation in high dimensions. Experiments demonstrate state-of-the-art privacy–utility trade-offs: on MNIST with $varepsilon = 8$, AdaDPIGU achieves 99.12% test accuracy—nearly matching the non-private baseline; on CIFAR-10 with $varepsilon = 4$, it attains 73.21% accuracy, surpassing the non-private baseline. These results validate AdaDPIGU’s effectiveness in preserving utility under stringent privacy constraints.

Technology Category

• Machine Learning: Privacy
• Computer Vision: Adversarial Attacks & Robustness
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Security and Privacy: Privacy-enhancing technologies
• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

Differential privacy has been proven effective for stochastic gradient descent; however, existing methods often suffer from performance degradation in high-dimensional settings, as the scale of injected noise increases with dimensionality. To tackle this challenge, we propose AdaDPIGU--a new differentially private SGD framework with importance-based gradient updates tailored for deep neural networks. In the pretraining stage, we apply a differentially private Gaussian mechanism to estimate the importance of each parameter while preserving privacy. During the gradient update phase, we prune low-importance coordinates and introduce a coordinate-wise adaptive clipping mechanism, enabling sparse and noise-efficient gradient updates. Theoretically, we prove that AdaDPIGU satisfies $(varepsilon, δ)$-differential privacy and retains convergence guarantees. Extensive experiments on standard benchmarks validate the effectiveness of AdaDPIGU. All results are reported under a fixed retention ratio of 60%. On MNIST, our method achieves a test accuracy of 99.12% under a privacy budget of $ε= 8$, nearly matching the non-private model. Remarkably, on CIFAR-10, it attains 73.21% accuracy at $ε= 4$, outperforming the non-private baseline of 71.12%, demonstrating that adaptive sparsification can enhance both privacy and utility.

Problem

• Reduces performance degradation in high-dimensional private SGD
• Enhances privacy and utility via adaptive gradient clipping
• Improves accuracy with importance-based sparse gradient updates

Innovation

• Adaptive clipping for noise-efficient updates
• Importance-based gradient pruning for sparsity
• Differentially private Gaussian pretraining mechanism