Computation-Utility-Privacy Tradeoffs in Bayesian Estimation

2026-03-18

AI Summary

This work addresses the challenge of simultaneously achieving computational efficiency, statistical optimality, and differential privacy in high-dimensional Bayesian estimation. Focusing on Gaussian mean estimation and linear regression, it proposes the first computationally efficient differentially private algorithm that attains near-Bayes-optimal mean squared error under a Gaussian prior. By introducing a novel framework that translates privacy guarantees into robustness properties and incorporating new constraints based on short-flat decompositions, the authors extend the Sum-of-Squares (SoS) method to construct robust estimators for non-robust objectives such as empirical means and ordinary least squares. The resulting algorithm achieves a $(1+o(1))$ multiplicative factor over the Bayes-optimal error in both settings and is provably computationally optimal within the low-degree polynomial model, substantially outperforming existing efficient approaches.

Technology Category

• Machine Learning: Privacy
• Search and Optimization: Non-convex Optimization
• Intelligent Robots: State Estimation

Application Category

• Security and Privacy: Large-scale security measurements
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

Bayesian methods lie at the heart of modern data science and provide a powerful scaffolding for estimation in data-constrained settings and principled quantification and propagation of uncertainty. Yet in many real-world use cases where these methods are deployed, there is a natural need to preserve the privacy of the individuals whose data is being scrutinized. While a number of works have attempted to approach the problem of differentially private Bayesian estimation through either reasoning about the inherent privacy of the posterior distribution or privatizing off-the-shelf Bayesian methods, these works generally do not come with rigorous utility guarantees beyond low-dimensional settings. In fact, even for the prototypical tasks of Gaussian mean estimation and linear regression, it was unknown how close one could get to the Bayes-optimal error with a private algorithm, even in the simplest case where the unknown parameter comes from a Gaussian prior. In this work, we give the first efficient algorithms for both of these problems that achieve mean-squared error $(1+o(1))\mathrm{OPT}$ and additionally show that both tasks exhibit an intriguing computational-statistical gap. For Bayesian mean estimation, we prove that the excess risk achieved by our method is optimal among all efficient algorithms within the low-degree framework, yet is provably worse than what is achievable by an exponential-time algorithm. For linear regression, we prove a qualitatively similar lower bound. Our algorithms draw upon the privacy-to-robustness framework of arXiv:2212.05015, but with the curious twist that to achieve private Bayes-optimal estimation, we need to design sum-of-squares-based robust estimators for inherently non-robust objects like the empirical mean and OLS estimator. Along the way we also add to the sum-of-squares toolkit a new kind of constraint based on short-flat decompositions.

Problem

• Bayesian estimation
• differential privacy
• computation-utility tradeoff
• statistical-computational gap
• private algorithms

Innovation

• differentially private Bayesian estimation
• computational-statistical gap
• sum-of-squares
• privacy-robustness framework
• Bayes-optimal error