Spectral and Temporal Denoising for Differentially Private Optimization

2025-05-07

AI Summary

To address the significant performance degradation caused by noise injection in differentially private stochastic gradient descent (DP-SGD), this paper proposes the FFT-enhanced Kalman Filter (FFTKF), which improves private optimization efficacy while strictly satisfying (ε, δ)-differential privacy. Methodologically, we introduce a novel spectral-temporal joint denoising framework: fast Fourier transform (FFT) is employed for frequency-domain analysis, and a high-frequency mask is designed to actively steer noise into low-informative frequency bands; combined with a scalar-gain Kalman filter leveraging finite-difference Hessian approximation, gradient noise is dynamically suppressed along the temporal dimension. Extensive experiments across benchmarks—from MNIST to Tiny-ImageNet—and architectures—including CNN, Wide ResNet (WRN), and Vision Transformer (ViT)—demonstrate that FFTKF consistently outperforms DP-SGD and DiSK. Theoretically, we prove that FFTKF preserves the original privacy guarantee and achieves superior privacy–utility trade-offs.

Technology Category

• Machine Learning: Privacy
• Search and Optimization: Non-convex Optimization
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Security and Privacy: Privacy-enhancing technologies
• Responsible Web: Data and user privacy-enhancing technologies for the Web
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

This paper introduces the FFT-Enhanced Kalman Filter (FFTKF), a differentially private optimization method that addresses the challenge of preserving performance in DP-SGD, where added noise typically degrades model utility. FFTKF integrates frequency-domain noise shaping with Kalman filtering to enhance gradient quality while preserving $(varepsilon, delta)$-DP guarantees. It employs a high-frequency shaping mask in the Fourier domain to concentrate differential privacy noise in less informative spectral components, preserving low-frequency gradient signals. A scalar-gain Kalman filter with finite-difference Hessian approximation further refines the denoised gradients. With a per-iteration complexity of $mathcal{O}(d log d)$, FFTKF demonstrates improved test accuracy over DP-SGD and DiSK across MNIST, CIFAR-10, CIFAR-100, and Tiny-ImageNet datasets using CNNs, Wide ResNets, and Vision Transformers. Theoretical analysis confirms that FFTKF maintains equivalent privacy guarantees while achieving a tighter privacy-utility trade-off through reduced noise and controlled bias.

Problem

• Enhance gradient quality in DP-SGD while preserving privacy
• Reduce noise impact by shaping it in frequency domain
• Maintain privacy guarantees with improved model utility

Innovation

• FFT-Enhanced Kalman Filter for DP optimization
• Frequency-domain noise shaping preserves gradients
• Scalar-gain Kalman filter refines denoised gradients