DP-IVON-Gradsq: Differentially Private Squared-Gradient Improved Variational Online Newton

2026-07-26

AI Summary

This work addresses the challenge of effectively modeling uncertainty in Bayesian deep learning under differential privacy constraints, where privacy-preserving noise can interfere with the stochasticity inherent in posterior sampling. To resolve this issue, the authors propose DP-IVON-Gradsq, a novel method that integrates differential privacy into an improved variational online Newton optimizer (IVON). By introducing a noise-corrected gradient-squared estimator, the approach accurately captures curvature information, thereby decoupling privacy-induced noise from Bayesian sampling noise. Empirical evaluations on CIFAR-10 demonstrate that DP-IVON-Gradsq matches or outperforms established baselines such as DP-SGD and DP-Adam under weak to moderate privacy budgets (i.e., relatively large ε), confirming its efficacy in jointly preserving privacy and enabling principled Bayesian inference.

Abstract

Differential privacy provides formal privacy guarantees for training neural networks on sensitive data, while Bayesian deep learning offers a principled framework for uncertainty-aware prediction. Combining these two objectives remains challenging, as privacy noise can interact with the stochasticity introduced by Bayesian posterior sampling. In this work, we investigate differentially private variational Bayesian learning through the Improved Variational Online Newton (IVON) optimizer. We introduce DP-IVON-Gradsq, a private variant of IVON. The proposed method constructs its curvature estimate from the privatized gradient using a noise-corrected squared-gradient estimator, reducing the direct interaction between posterior-sampling noise and privacy noise while preserving the Adam-like computational efficiency of IVON. We evaluate DP-IVON-Gradsq on CIFAR-10 against the standard private optimizers DP-SGD and DP-Adam over a range of privacy budgets. The results show that DP-IVON-Gradsq is competitive under weak-to-moderate privacy constraints, i.e., large-to-moderate values of $\varepsilon$, while degrading under strong privacy. Code is available at https://github.com/NourJamoussi/DP-IVON-Gradsq.git.

Problem

• differential privacy
• Bayesian deep learning
• privacy-preserving machine learning
• uncertainty-aware prediction
• neural network training

Innovation

• Differential Privacy
• Variational Bayesian Learning
• IVON
• Squared-Gradient Estimator
• Private Optimization