🤖 AI Summary
Diffusion models (DMs) achieve state-of-the-art performance in text-to-image generation, yet their high memory footprint and computational cost—stemming from iterative sampling—hinder edge deployment. Knowledge distillation of pre-trained DMs has emerged as a key efficiency-enhancement strategy, but existing works lack systematic organization. This paper presents the first methodology-driven, structured survey of DM distillation, categorizing approaches into three paradigms: output-loss distillation, trajectory distillation, and adversarial distillation. We unify their formulations by integrating techniques from knowledge distillation, denoising trajectory fitting, adversarial training, and multi-step sampling approximation, clarifying underlying principles, boundaries, interconnections, and application scopes. Our analysis identifies fundamental challenges—including the sampling-fidelity trade-off and cross-paradigm integration—and proposes future directions such as scalable distillation frameworks. This work fills a critical gap by providing the first comprehensive, taxonomy-based review of DM distillation.
📝 Abstract
Diffusion Models~(DMs) have emerged as the dominant approach in Generative Artificial Intelligence (GenAI), owing to their remarkable performance in tasks such as text-to-image synthesis. However, practical DMs, such as stable diffusion, are typically trained on massive datasets and thus usually require large storage. At the same time, many steps may be required, i.e., recursively evaluating the trained neural network, to generate a high-quality image, which results in significant computational costs during sample generation. As a result, distillation methods on pre-trained DM have become widely adopted practices to develop smaller, more efficient models capable of rapid, few-step generation in low-resource environment. When these distillation methods are developed from different perspectives, there is an urgent need for a systematic survey, particularly from a methodological perspective. In this survey, we review distillation methods through three aspects: output loss distillation, trajectory distillation and adversarial distillation. We also discuss current challenges and outline future research directions in the conclusion.