On the Memorization of Consistency Distillation for Diffusion Models

2026-04-26

AI Summary

This work investigates the impact of consistency distillation on memorization behavior in diffusion models, with a focus on how student models inherit or suppress memorized patterns when the teacher model has already memorized training data. Combining theoretical analysis based on neural networks with random features and systematic empirical experiments, the study reveals for the first time that consistency distillation not only accelerates inference but also actively suppresses unstable feature directions associated with memorization. The results demonstrate that this approach substantially reduces memory transfer while preserving or even enhancing sample generation quality, thereby effectively improving the trade-off between memorization and generalization.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Other Foundations of Machine Learning
• Search and Optimization: Learning to Search

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Diffusion models are central to modern generative modeling, and understanding how they balance memorization and generalization is critical for reliable deployment. Recent work has shown that memorization in diffusion models is shaped by training dynamics, with generalization and memorization emerging at different stages of training. However, deployed diffusion models are often further distilled, introducing an additional training phase whose impact on memorization is not well understood. In this work, we analyze how distillation reshapes memorization behavior in diffusion models, taking consistency distillation as a representative framework. Empirically, we show that when applied to a teacher model that has memorized data, consistency distillation significantly reduces transferred memorization in the student while preserving, and sometimes improving, sample quality. To explain this behavior, we provide a theoretical analysis using a random feature neural network model [Bonnaire et al., 2025], showing that consistency distillation suppresses unstable feature directions associated with memorization while preserving stable, generalizable modes. Our findings suggest that distillation can serve not only as an acceleration tool, but also as a mechanism for improving the memorization-generalization trade-off.

Problem

• memorization
• diffusion models
• consistency distillation
• generalization
• model distillation

Innovation

• consistency distillation
• memorization
• diffusion models
• generalization
• feature stability