Privacy for Free in the Over-Parameterized Regime

2024-10-18

arXiv.org

AI Summary

Differential privacy gradient descent (DP-GD) suffers from significant generalization degradation in overparameterized deep learning, leading to the prevailing belief that overparameterization inherently harms private learning performance. Method: We analyze DP-GD under a random features model with quadratic loss, leveraging high-dimensional statistical analysis integrated with differential privacy theory. Contribution/Results: We rigorously prove that when the number of parameters $p gg n$ (sample size) and strong privacy is enforced ($varepsilon = o(1)$), the generalization error of DP-GD vanishes asymptotically—i.e., achieves $o(1)$ population risk $|R_P|$. This is the first formal demonstration that overparameterization need not incur privacy-accuracy trade-offs; rather, it enables *nearly cost-free privacy* under stringent privacy constraints. Our result overturns the conventional wisdom that overparameterization inevitably degrades private learning performance. Beyond theoretical novelty, this work establishes a new paradigm for private deep learning and provides rigorous feasibility guarantees for efficient privacy-preserving training of large-scale models.

Technology Category

• Machine Learning: Privacy
• Natural Language Processing: Learning & Optimization for NLP
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Security and Privacy: Data transparency and provenance
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

Differentially private gradient descent (DP-GD) is a popular algorithm to train deep learning models with provable guarantees on the privacy of the training data. In the last decade, the problem of understanding its performance cost with respect to standard GD has received remarkable attention from the research community, which formally derived upper bounds on the excess population risk $R_{P}$ in different learning settings. However, existing bounds typically degrade with over-parameterization, i.e., as the number of parameters $p$ gets larger than the number of training samples $n$ -- a regime which is ubiquitous in current deep-learning practice. As a result, the lack of theoretical insights leaves practitioners without clear guidance, leading some to reduce the effective number of trainable parameters to improve performance, while others use larger models to achieve better results through scale. In this work, we show that in the popular random features model with quadratic loss, for any sufficiently large $p$, privacy can be obtained for free, i.e., $left|R_{P} ight| = o(1)$, not only when the privacy parameter $varepsilon$ has constant order, but also in the strongly private setting $varepsilon = o(1)$. This challenges the common wisdom that over-parameterization inherently hinders performance in private learning.

Problem

• Examines performance cost of differentially private gradient descent (DP-GD) vs standard GD
• Challenges over-parameterization hindering private learning performance
• Shows privacy can be free in random features model with quadratic loss

Innovation

• DP-GD ensures privacy in overparameterized models
• Privacy achieved for free in large models
• Over-parameterization does not hinder private learning