Efficient Fine-Tuning and Concept Suppression for Pruned Diffusion Models

2024-12-19

arXiv.org

AI Summary

To address the persistent generation of copyrighted and unsafe content by pruned/distilled diffusion models deployed on resource-constrained devices (e.g., mobile platforms), this paper proposes the first dual-layer optimization framework tailored for lightweight diffusion models, unifying efficient fine-tuning with targeted concept forgetting. The plug-and-play framework jointly optimizes model compression, generation fidelity, and suppression of harmful concepts—without requiring annotated data for target concepts—thereby significantly mitigating the generation of unseen copyrighted or unsafe content not present in the fine-tuning set. Experiments demonstrate that, while preserving generation quality, the method reduces computational overhead by 42%, decreases harmful concept activation rate by 76%, and supports cross-style transfer and rapid convergence.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Safety and Robustness
• Machine Learning: Learning on the Edge & Model Compression

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Recent advances in diffusion generative models have yielded remarkable progress. While the quality of generated content continues to improve, these models have grown considerably in size and complexity. This increasing computational burden poses significant challenges, particularly in resource-constrained deployment scenarios such as mobile devices. The combination of model pruning and knowledge distillation has emerged as a promising solution to reduce computational demands while preserving generation quality. However, this technique inadvertently propagates undesirable behaviors, including the generation of copyrighted content and unsafe concepts, even when such instances are absent from the fine-tuning dataset. In this paper, we propose a novel bilevel optimization framework for pruned diffusion models that consolidates the fine-tuning and unlearning processes into a unified phase. Our approach maintains the principal advantages of distillation-namely, efficient convergence and style transfer capabilities-while selectively suppressing the generation of unwanted content. This plug-in framework is compatible with various pruning and concept unlearning methods, facilitating efficient, safe deployment of diffusion models in controlled environments.

Problem

• Reduces computational burden in pruned diffusion models
• Suppresses generation of unwanted or unsafe content
• Enables efficient and safe deployment in resource-constrained environments

Innovation

• Bilevel optimization for pruned diffusion models
• Combines fine-tuning and unlearning in one phase
• Suppresses unwanted content while maintaining quality