formal privacy definitions

Formally specifying privacy and incentive properties (e.g., failure privacy, identifiability, routing anonymity), modeling adversarial distinguishing power, and proving that system designs satisfy the required privacy guarantees and incentive constraints.

formalprivacydefinitions

12-Month Skill Trend

Momentum and market value over time
Trending
Score
+20 in 12 mo
96
12 mo agoNow
Career
Value
+$12K in 12 mo
$42K/year
12 mo agoNow

Recommended Survey Paper

Quick overview of the field
View more

Must-Read Papers

Most classic and influential ideas
View more

This work addresses the challenge of precisely characterizing the overall privacy guarantee when composing mechanisms under multiple heterogeneous differential privacy (DP) constraints. The authors propose a general composition framework that, for the first time, enables an exact description of the resulting privacy region after composing an arbitrary number of mechanisms subject to diverse DP bounds. By constructing a binary hypothesis testing–based mixture model and integrating probabilistic mixing with f-DP approximation techniques, the framework yields an exact composition theorem for multiple DP constraints. Moreover, the approach naturally extends to the f-DP setting, significantly enhancing both the tightness and applicability of compositional privacy analysis.

compositiondifferential privacyf-DP

Persuasive Privacy

Jan 30, 2026

This work addresses a critical limitation in existing privacy definitions, which often overlook purpose-driven requirements and struggle to accommodate deterministic algorithms. To overcome this, the authors propose a unified privacy quantification framework grounded in Bayesian game theory. This framework subsumes both pure and approximate differential privacy as special cases and, for the first time, provides formal privacy guarantees for deterministic algorithms. By explicitly modeling the adversary’s utility and posterior beliefs, the approach naturally yields a novel interpretation of post-processing invariance. The resulting framework not only encompasses mainstream privacy notions but also extends applicability to a broader class of algorithms, significantly enhancing both the theoretical expressiveness and practical relevance of formal privacy guarantees.

Bayesian game theorydeterministic algorithmsdifferential privacy

How to Get Actual Privacy and Utility from Privacy Models: the k-Anonymity and Differential Privacy Families

Oct 13, 2025
JD
Josep Domingo-Ferrer
🏛️ Universitat Rovira i Virgili | CYBERCAT-Center for Cybersecurity Research of Catalonia

Existing mainstream privacy models suffer from fundamental limitations: k-anonymity operates syntactically, rendering it vulnerable to background knowledge attacks and lacking semantic constraints; differential privacy faces a sharp utility–privacy trade-off—small privacy budgets cause severe data distortion, while large budgets degrade privacy guarantees. Method: We propose Semantic k-Anonymity, which formally incorporates domain-specific semantic constraints and dependencies among sensitive attributes to reconstruct the equivalence-class partitioning mechanism—enhancing disclosure resistance without compromising data utility. Contribution/Results: Through rigorous formal modeling, principled semantic constraint design, and empirical risk assessment, we demonstrate that Semantic k-Anonymity achieves more robust privacy protection and higher data utility than conventional k-anonymity and differential privacy in realistic settings, thereby reducing reliance on post-hoc risk evaluation.

Current privacy models require costly empirical risk assessmentsDifferential privacy causes excessive utility loss with small budgetsk-anonymity fails to fully prevent data disclosure risks

Contextually Private Mechanisms

Dec 20, 2021
AH
A. Haupt
🏛️ Stanford Digital Economy Lab | Harvard Society of Fellows

This paper addresses dynamic information elicitation in mechanism design, where excessive contextual information leakage from agents’ private data may occur. Method: We introduce “contextual privacy”—a formal privacy notion requiring mechanisms to elicit only the minimal private information necessary to determine the outcome, thereby avoiding unnecessary contextual correlations. We formally define contextual privacy and characterize its necessary and sufficient conditions under the sequential identification assumption. Contribution/Results: We prove that canonical mechanisms—including serial dictatorship and first-price auctions—satisfy contextual privacy, whereas most k-price auctions and stable matching mechanisms do not. Furthermore, under an extended anonymity-based counting assumption, we construct contextual-privacy-preserving implementations for several non-contextually-private mechanisms, substantially broadening the class of mechanisms admitting strong privacy guarantees. Our work unifies mechanism design, privacy theory, and sequential information elicitation, establishing a new paradigm for privacy-sensitive economic mechanisms.

Designing dynamic protocols to elicit private information while minimizing contextual privacy violations.Ensuring maximal contextual privacy in auctions by strategically ordering queries to protect bidders.Implementing choice rules where agents are collectively pivotal without unnecessary individual disclosures.

$Proovarphi$: A ZKP Market Mechanism

Apr 09, 2024
WW
Wenhao Wang
🏛️ Yale University | Hebrew University of Jerusalem | Matter Labs | Imperial College London

Zero-knowledge proof (ZKP) generation incurs substantial computational overhead, necessitating efficient outsourcing mechanisms for applications such as ZK-Rollups; however, existing markets lack formal modeling and incentive guarantees. Method: We propose the first formal ZKP market model characterizing the interaction between users submitting proof tasks and provers bidding to generate proofs. We design $Proovarphi$, an incentive-compatible and budget-balanced auction mechanism inspired by the Vickrey–Clarke–Groves (VCG) framework and integrated with cryptographic protocols. To enhance robustness, we systematically incorporate defenses against Sybil attacks, capacity overbidding, and collusion. Contribution/Results: We formally prove $Proovarphi$’s incentive compatibility and budget balance. Analysis demonstrates its effectiveness in mitigating multiple real-world security threats. Our work establishes the first deployable, robust market infrastructure for ZKP outsourcing, bridging a critical gap between theory and practice in verifiable computation markets.

Addresses practical challenges like Sybil attacks and prover capacity misreporting.Designs a formal ZKP market model for user-prover interactions.Introduces an auction-based mechanism ensuring incentive compatibility and budget balance.

Latest Papers

What's happening recently
View more

This work addresses the absence of a unified formal foundation for systematically evaluating the security of access control and permission mechanisms in language model–based agents. It extends cryptographic formal methods to the domain of AI agents by introducing the AIOracle formal model and a security game framework encompassing confidentiality, integrity, and availability. The paper further establishes a taxonomy of agent-specific attacks and, through modular decomposition of beneficial and harmless objectives, constructs provably secure reductions that reveal a fundamental tension between training data confidentiality and system completeness. By establishing a quantifiable theoretical basis for AI agent security, this study demonstrates the necessity of modular design for achieving provable security and provides a formal verification pathway for future secure agent architectures.

access controlAI agentsformal foundations

Composition Theorems for f-Differential Privacy

Dec 23, 2025
NF
Natasha Fernandes
🏛️ Macquarie University | UNSW

This work addresses the looseness of privacy loss estimation under $f$-differential privacy (fDP) in complex composition scenarios. We propose a novel compositional analysis framework grounded in statistical hypothesis testing and quantitative information flow (QIF) channel models. First, we establish a Galois connection between fDP and QIF, revealing their fundamental equivalence under the hypothesis-testing semantics. Leveraging this connection, we derive a general composition theorem that is both broadly applicable and significantly tighter than existing bounds. Our method substantially improves the precision of compositional analysis for canonical mechanisms—particularly the Gaussian mechanism—yielding more accurate and computationally tractable privacy loss predictions. By bridging theoretical fDP characterizations with practical budget management, this framework advances the operational deployment of fDP in real-world privacy-preserving systems.

Enables improved analysis of complex privacy designsEstablishes equivalence between fDP and Quantitative Information FlowProves composition theorems for f-differential privacy mechanisms

Existing formal methods struggle to verify the privacy guarantees of modern differential privacy libraries that employ complex programming patterns such as higher-order functions, local state, and interactive algorithms. This work proposes a probabilistic higher-order separation logic that, for the first time, incorporates first-class support for privacy budgets within separation logic, treating them as composable resources to enable modular reasoning. Implemented in the Rocq proof assistant, the logic successfully verifies differential privacy programs featuring higher-order combinators, caching, and interactive mechanisms. Furthermore, the authors construct a formally verified library of mechanisms, including the online sparse vector technique and privacy filters inspired by OpenDP, thereby enabling end-to-end verification of client programs.

differential privacyhigher-order functionsmodular verification

SLIP-SEC: Formalizing Secure Protocols for Model IP Protection

Oct 28, 2025
RJ
Racchit Jain
🏛️ Microsoft

To address model theft and intellectual property (IP) leakage risks when deploying large language models (LLMs) on partially trusted or untrusted devices, this paper proposes SLIP—the first formal framework enabling information-theoretically secure hybrid inference. Methodologically, SLIP decomposes model weight matrices additively, integrating random masking with probabilistic verification to enable collaborative inference across trusted and untrusted environments. We formally prove that the protocol achieves strict security under both honest-but-curious and malicious adversary models, while preserving computational efficiency. Key contributions include: (1) pioneering the application of information-theoretic security to LLM inference protection; (2) establishing a provably secure model decomposition scheme with well-defined execution semantics; and (3) designing a lightweight protocol that simultaneously ensures practical deployability and strong security guarantees. Experimental evaluation confirms SLIP’s efficacy in mitigating IP leakage without significant overhead.

Achieves information-theoretic security against honest-but-curious and malicious adversariesFormalizes secure inference protocols to protect LLM intellectual property from theftSplits model computation between trusted and untrusted resources with security guarantees

This work addresses the limitations of existing privacy-preserving frameworks—such as differential privacy and statistical disclosure control—in contextual adaptability and semantic rigor, which often hinder the provision of concrete and interpretable guarantees. The paper proposes a novel privacy quantification approach grounded in Bayesian decision theory, advocating for disclosure decisions based on a priori perspectives rather than data-conditional viewpoints. By introducing a context-aware and semantically precise definition of Bayesian adversarial privacy, the method transcends the rigid assumptions inherent in traditional differential privacy and establishes a more rigorous formal framework. Through adversarial modeling, prior-based inference, and concrete computational examples, both theoretical analysis and illustrative case studies demonstrate that the proposed approach offers significant advantages in semantic expressiveness and practical feasibility for privacy preservation.

Bayesian decision theorycontextual privacydifferential privacy

Hot Scholars

SZ

Shuning Zhang

Tsinghua University
HCIUsable Privacy and SecurityAI
CS

Cristiana Santos

Utrecht University
Compliance with Data Protection LawDark PatternsTracking
AH

Alexis Hiniker

Associate Professor, University of Washington
Human-Computer Interaction
VM

Victor Morel

Chalmers University of Technology
privacyinterplay law/technology
YY

Yaxing Yao

Assistant Professor at Johns Hopkins
PrivacyIoTsHCI