Public Diffusion Models, Private Images: Key-Controlled Inversion for Conditional Reconstruction

2026-06-22

AI Summary

This work addresses the privacy vulnerability in diffusion models under white-box settings, where intermediate latent representations can be inverted to recover the original input image. To mitigate this risk, the authors propose a key-controlled inversion framework that injects key-dependent noise during the inversion process. Leveraging the exponential error amplification inherent in diffusion dynamics, the method ensures that only authorized users possessing the correct key can accurately reconstruct the image. This approach establishes the first key-based access control mechanism in white-box scenarios, repurposing the model’s error propagation behavior as a security feature. The scheme provides provable IND-CPA security, and empirical evaluations demonstrate its robustness across diverse models and datasets, controllable reconstruction fidelity, and an adversary’s success probability that decays exponentially with the security parameter.

Abstract

Diffusion models are often deployed in settings where model parameters are publicly accessible (e.g., open-source libraries or released checkpoints). This white-box scenario creates a serious security risk: any user who obtains an intermediate latent representation can invert the process to recover the original input image. Most prior work on access control for generative models assumes a black-box model (i.e., parameters are kept secret), typically under an honest-but-curious adversary. By contrast, we address the more challenging and realistic white-box setting where all parameters are public. We present a key-controlled inversion framework that turns the inherent error propagation of diffusion models, which exponentially amplifies small perturbations, into a security asset. By injecting key-dependent noise into the inversion formula, we ensure that only a user with the correct key can reconstruct the original image; any other key yields unrecognizable output. Theoretically, by leveraging existing error-propagation theory for diffusion models, we prove that the resulting ciphertext distribution is IND-CPA secure and derive that the adversary's advantage is exponentially small in a tunable security parameter, hence negligible for any probabilistic polynomial-time (PPT) adversary. Experimentally, we validate these security guarantees across several models and datasets and further demonstrate cross-model robustness, that the injected key noise does not amplify the performance drop caused by model discrepancies.

Problem

• diffusion models
• white-box setting
• image inversion
• privacy protection
• access control

Innovation

• key-controlled inversion
• diffusion models
• white-box security
• error propagation
• IND-CPA security