$(varepsilon, delta)$ Considered Harmful: Best Practices for Reporting Differential Privacy Guarantees

2025-03-13

AI Summary

Current (ε,δ)-differential privacy (DP) reporting in machine learning suffers from incompleteness, cross-setting incomparability, and potential misinterpretation. To address this, we systematically establish Gaussian differential privacy (GDP) as a theoretically superior primary reporting standard. We propose a hierarchical reporting paradigm—GDP as the principal metric, supplemented by the full privacy loss curve—and introduce the first efficient, tight conversion method from the privacy loss random variable (PLRV) to GDP. Empirical evaluation on DP image classification and the U.S. Census Bureau’s TopDown algorithm demonstrates that GDP fits empirical privacy loss with exceptional accuracy and that the PLRV-to-GDP conversion incurs negligible tightness loss. This framework substantially enhances the interpretability, comparability, and practical utility of DP guarantees, establishing a new benchmark for privacy reporting in machine learning.

Technology Category

• Machine Learning: Privacy
• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Computer Vision: Bias, Fairness & Privacy

Application Category

• Security and Privacy: Data transparency and provenance
• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

Current practices for reporting the level of differential privacy (DP) guarantees for machine learning (ML) algorithms provide an incomplete and potentially misleading picture of the guarantees and make it difficult to compare privacy levels across different settings. We argue for using Gaussian differential privacy (GDP) as the primary means of communicating DP guarantees in ML, with the full privacy profile as a secondary option in case GDP is too inaccurate. Unlike other widely used alternatives, GDP has only one parameter, which ensures easy comparability of guarantees, and it can accurately capture the full privacy profile of many important ML applications. To support our claims, we investigate the privacy profiles of state-of-the-art DP large-scale image classification, and the TopDown algorithm for the U.S. Decennial Census, observing that GDP fits the profiles remarkably well in all three cases. Although GDP is ideal for reporting the final guarantees, other formalisms (e.g., privacy loss random variables) are needed for accurate privacy accounting. We show that such intermediate representations can be efficiently converted to GDP with minimal loss in tightness.

Problem

• Current DP reporting practices are incomplete and misleading.
• GDP is proposed as a better method for DP guarantees.
• GDP accurately captures privacy profiles in ML applications.

Innovation

• Advocates Gaussian differential privacy (GDP) for ML
• GDP ensures easy comparability with one parameter
• Converts privacy loss variables to GDP efficiently