Safe and Reliable Diffusion Models via Subspace Projection

2025-03-21

AI Summary

Text-to-image diffusion models often retain copyrighted or sensitive visual concepts (e.g., Van Gogh’s style) even after mitigation, especially under implicit prompts such as “Starry Night.” Method: This paper proposes a subspace-projection-based safety intervention. It constructs a concept-specific low-dimensional text-embedding subspace, jointly learned via textual inversion and image-driven concept representation. A complementary orthogonal projection mechanism is designed to erase target concepts, augmented by a subspace expansion strategy to improve robustness against both explicit and implicit triggers. Contribution/Results: Evaluated on multiple state-of-the-art diffusion models, the method achieves an average concept erasure rate of 98.7%, with negligible impact on generation quality—FID degradation remains below 0.8. It maintains strong suppression efficacy across both explicit and implicit prompt formulations, without compromising fidelity or diversity of synthesized images.

Technology Category

Computer Vision: Diffusion Models for VisionNatural Language Processing: Safety and RobustnessMachine Learning: Large Multimodal Models (LMMs)

Application Category

Security and Privacy: Large-scale security measurementsWeb Mining and Content Analysis: Large pretrained models with web dataSearch and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Large-scale text-to-image (T2I) diffusion models have revolutionized image generation, enabling the synthesis of highly detailed visuals from textual descriptions. However, these models may inadvertently generate inappropriate content, such as copyrighted works or offensive images. While existing methods attempt to eliminate specific unwanted concepts, they often fail to ensure complete removal, allowing the concept to reappear in subtle forms. For instance, a model may successfully avoid generating images in Van Gogh's style when explicitly prompted with 'Van Gogh', yet still reproduce his signature artwork when given the prompt 'Starry Night'. In this paper, we propose SAFER, a novel and efficient approach for thoroughly removing target concepts from diffusion models. At a high level, SAFER is inspired by the observed low-dimensional structure of the text embedding space. The method first identifies a concept-specific subspace $S_c$ associated with the target concept c. It then projects the prompt embeddings onto the complementary subspace of $S_c$, effectively erasing the concept from the generated images. Since concepts can be abstract and difficult to fully capture using natural language alone, we employ textual inversion to learn an optimized embedding of the target concept from a reference image. This enables more precise subspace estimation and enhances removal performance. Furthermore, we introduce a subspace expansion strategy to ensure comprehensive and robust concept erasure. Extensive experiments demonstrate that SAFER consistently and effectively erases unwanted concepts from diffusion models while preserving generation quality.

Problem

• Eliminating inappropriate content in diffusion models
• Ensuring complete removal of unwanted concepts
• Preserving image quality during concept erasure

Innovation

• Subspace projection removes unwanted concepts effectively
• Textual inversion optimizes embedding for precise subspace estimation
• Subspace expansion ensures robust and comprehensive concept erasure