Differentially Private Adaptation of Diffusion Models via Noisy Aggregated Embeddings

2024-11-22

arXiv.org

AI Summary

Diffusion models face significant challenges in style and content transfer under differential privacy (DP) constraints—conventional DP-SGD incurs prohibitive computational and memory overhead on large models, while excessive noise severely degrades fidelity on small private datasets. Method: We propose a lightweight, weight-free DP adaptation framework that integrates Textual Inversion with DP for the first time. Our approach injects Laplacian or Gaussian noise into the embedding space, enforces embedding clipping and aggregation, and optimizes privacy budget allocation across training steps. Contribution/Results: Evaluated on two private datasets—single-artist artwork and Paris 2024 Olympic logos—our method achieves high-fidelity private adaptation with ε = 2–8. Under strong privacy guarantees, it significantly outperforms DP-SGD baselines in visual quality of style transfer, preserving semantic clarity and structural coherence. This breaks the performance bottleneck of conventional DP fine-tuning for large diffusion models trained on limited private data.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Privacy
• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy

Application Category

• Security and Privacy: Privacy-enhancing technologies
• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

We introduce a novel method for adapting diffusion models under differential privacy (DP) constraints, enabling privacy-preserving style and content transfer without fine-tuning model weights. Traditional approaches to private adaptation, such as DP-SGD, incur significant computational and memory overhead when applied to large, complex models. In addition, when adapting to small-scale specialized datasets, DP-SGD incurs large amount of noise that significantly degrades the performance. Our approach instead leverages an embedding-based technique derived from Textual Inversion (TI) and adapted with differentially private mechanisms. We apply TI to Stable Diffusion for style adaptation using two private datasets: a collection of artworks by a single artist and pictograms from the Paris 2024 Olympics. Experimental results show that the TI-based adaptation achieves superior fidelity in style transfer, even under strong privacy guarantees.

Problem

• Adapt diffusion models with privacy
• Reduce computational and memory overhead
• Enhance style transfer performance privately

Innovation

• Differential privacy for diffusion models
• Embedding-based technique adaptation
• Noise reduction in style transfer