🤖 AI Summary
To address the fundamental trade-off between privacy preservation and utility in high-dimensional, multi-feature data, this paper proposes a coordinate-wise independent but non-identically distributed (i.n.i.d.) noise mechanism. We provide the first theoretical proof that the Laplace mechanism can strictly outperform the Gaussian mechanism in high dimensions under differential privacy. By explicitly modeling heterogeneous privacy sensitivities across coordinates, we derive formal privacy guarantees for i.n.i.d. noise and establish an optimization framework for designing optimal noise parameters—supporting both weighted mean-squared error (MSE) and ℓₚ norm error minimization. The method is instantiated in private coordinate descent, differentially private PCA, and private deep learning with group-wise gradient clipping. Empirical evaluation demonstrates consistent utility gains over state-of-the-art baselines under identical privacy budgets across optimization, dimensionality reduction, and model training tasks—challenging the conventional wisdom that Gaussian noise is inherently superior in high-dimensional settings.
📝 Abstract
Conventionally, in a differentially private additive noise mechanism, independent and identically distributed (i.i.d.) noise samples are added to each coordinate of the response. In this work, we formally present the addition of noise that is independent but not identically distributed (i.n.i.d.) across the coordinates to achieve tighter privacy-accuracy trade-off by exploiting coordinate-wise disparity in privacy leakage. In particular, we study the i.n.i.d. Gaussian and Laplace mechanisms and obtain the conditions under which these mechanisms guarantee privacy. The optimal choice of parameters that ensure these conditions are derived considering (weighted) mean squared and $ell_{p}^{p}$-errors as measures of accuracy. Theoretical analyses and numerical simulations demonstrate that the i.n.i.d. mechanisms achieve higher utility for the given privacy requirements compared to their i.i.d. counterparts. One of the interesting observations is that the Laplace mechanism outperforms Gaussian even in high dimensions, as opposed to the popular belief, if the irregularity in coordinate-wise sensitivities is exploited. We also demonstrate how the i.n.i.d. noise can improve the performance in private (a) coordinate descent, (b) principal component analysis, and (c) deep learning with group clipping.