diffusion modeling

The field of diffusion models for visual generation lacks systematic, pedagogically structured educational resources. Method: This project develops a teaching-oriented unified framework targeting undergraduate and graduate students, systematically integrating foundational probabilistic modeling—包括 forward diffusion, reverse denoising, stochastic differential equation (SDE) solvers, and score matching—with state-of-the-art conditional image and video generation. The framework emphasizes structured exposition of modeling principles, training paradigms, and sampling mechanisms to establish a clear, reproducible conceptual foundation. Contribution/Results: It significantly lowers the entry barrier for learners and fills a critical gap in introductory, comprehensive tutorials on diffusion models. The framework has become a widely adopted pedagogical benchmark and cross-disciplinary reference for both diffusion model instruction and applied research.

The boundary between memorization (reproducing training data) and generalization (generating novel samples) in diffusion models remains poorly understood, yet this boundary critically determines copyright and privacy risks. Method: We establish a theoretical framework that—first for underparameterized diffusion models—reveals the critical mechanism governing the memorization–generalization phase transition: the transition is governed by the relative weighting of memory and generalization components in the training loss. We derive an analytical criterion to precisely predict the critical model size at which memorization dominates, and design a “mathematical laboratory” using synthetic and structured images to quantify the evolution of loss component weights via both theoretical analysis and gradient descent experiments. Contribution/Results: Experiments validate the high accuracy of our theoretical predictions, yielding explicit model capacity thresholds beyond which memorization becomes likely. This work provides the first analytically tractable and empirically verifiable theoretical foundation for designing safe, controllable diffusion models.

Video diffusion models (VDMs) pose a risk of memorizing and regenerating training data—potentially leaking copyrighted content—yet this issue remains systematically unexplored in the video domain. This work formally defines *content memory* and *motion memory* as two orthogonal dimensions of memorization and introduces the first dedicated memory evaluation framework for VDMs. Our framework comprises: (i) decoupled quantitative metrics, (ii) a curated set of high-risk prompt templates, and (iii) a cross-modal (image + video) similarity assessment method operating at both frame-level and optical-flow-level, augmented by statistical significance testing and multi-model benchmarking. Experiments on leading open-source VDMs reveal pervasive cross-modal memorization behavior, including successful reconstruction of original training clips. We further propose a lightweight detection strategy enabling efficient identification of both content and motion memory. This work establishes foundational theory and practical tools for the safety evaluation of VDMs.

In diffusion model training, uniform timestep sampling ignores the variance heterogeneity of gradients across timesteps, rendering high-variance timesteps convergence bottlenecks. To address this, we propose the first online evaluation mechanism that dynamically assesses—per iteration—the impact of gradient updates on the objective function, enabling adaptive, non-uniform timestep sampling focused on optimization-sensitive timesteps. Our method transcends conventional static weighting or heuristic sampling by unifying gradient variance analysis, objective impact tracking, and importance sampling. It achieves principled, real-time timestep prioritization without requiring precomputed statistics or architectural modifications. Experiments across diverse datasets, noise schedules, and network architectures demonstrate consistent improvements: our approach accelerates convergence and enhances final model performance compared to state-of-the-art timestep sampling and weighting strategies.

Diffusion models trained on small-scale datasets often exhibit counterproductive memorization—reproducing training samples rather than learning the underlying data distribution, thereby impairing generalization. Method: We introduce the Effective Model Memory (EMM) metric, the largest training set size for which a model can still approach the theoretical optimal solution. Leveraging denoising score matching theory, controlled ablation experiments, and adversarial perturbation tests (e.g., random label assignment), we systematically dissect how data distribution, model architecture, and training dynamics jointly determine the memorization boundary. Contribution/Results: Our analysis reveals that anomalous conditions—particularly random labeling—strongly trigger memorization. EMM serves as a practical, quantifiable early-warning indicator of memorization risk. All code and experimental protocols are publicly released to ensure full reproducibility.

In diffusion model training, non-uniform timestep distribution causes excessive sampling in the convergence regime—where gradient signals are weak—severely degrading training efficiency. To address this, we propose a dynamic timestep analysis framework that, for the first time, identifies three distinct temporal regimes: acceleration, deceleration, and convergence. Based on this characterization, we design an asymmetric importance sampling strategy and a process-incremental weighting mechanism, enabling targeted optimization of training efficiency at the individual-timestep level. Our method is plug-and-play—requiring no modifications to model architecture or loss function. Extensive experiments across diverse diffusion models (DDPM, DDIM, Score SDE), datasets (CIFAR-10, CelebA, LSUN), and tasks (image generation, super-resolution) demonstrate an average 3.0× training speedup, accompanied by proportional reductions in hardware cost and energy consumption, with zero degradation in generation quality or downstream performance.

This work addresses the lack of a systematic theoretical understanding of how finite-sample learning, neural network parameterization, and numerical discretization jointly affect generation quality in diffusion models. The authors develop a unified framework for convergence and generalization analysis, decomposing the overall generation error— for the first time—into four interpretable components: forward truncation error, backward discretization error, generalization error (accounting for both data finiteness and forward discretization), and optimization gap. Leveraging a ResNet-type score estimator and combining tools from numerical analysis of stochastic differential equations with total variation distance bounds, they quantitatively characterize the joint influence of training sample size, temporal grid density, and optimization accuracy on generation fidelity, thereby establishing end-to-end theoretical guarantees.

This work addresses the limitation of static sampling parameters—such as prompts, classifier-free guidance scales, and noise schedules—in diffusion models, which fail to adapt to varying prompts and denoising stages. The authors propose LeSAMP, a novel framework that formulates sampling parameter scheduling as a reinforcement learning problem. LeSAMP leverages large language models to generate prompt-conditioned, timestep-adaptive parameter policies, optimized jointly via human preference models and a vision-language model acting as a judge (VLM-as-a-judge). Evaluated on Flux.1 [dev] and Stable Diffusion 3.5, LeSAMP significantly outperforms baseline methods, achieving a human preference win rate of 68.12%, a VLM-assessed score of 73.37%, and a user-study win rate of 59.46%, thereby establishing a new paradigm for dynamic optimization of diffusion sampling.

Diffusion models exhibit strong generalization capabilities even before fully memorizing their training data, challenging the conventional theoretical assumption that memorization and generalization are inherently at odds. This work shifts the focus from “why models do not memorize” to “what is learned during the pre-memorization phase,” proposing a new theoretical perspective that integrates model capacity, optimization-induced implicit regularization, and architectural inductive biases. Empirical analysis on CIFAR-10 demonstrates that diffusion models acquire essential structural features of the data well before memorization occurs, revealing a distinctive generalization mechanism. These findings lay the groundwork for developing novel generalization theories tailored to generative models and highlight several critical open questions to guide future breakthroughs in the field.

This work addresses the limited controllability of temporal dynamics and editing in existing video diffusion Transformer models. To overcome this, the authors propose a lightweight temporal control module that enables explicit and fine-grained manipulation of motion speed and temporal structure without altering the pre-trained DiT backbone. By effectively leveraging the generative priors learned during pre-training, the method significantly enhances temporal controllability in video generation while preserving the original output quality. The approach thus offers a practical and efficient solution for precise temporal editing in diffusion-based video synthesis.

This work investigates the impact of consistency distillation on memorization behavior in diffusion models, with a focus on how student models inherit or suppress memorized patterns when the teacher model has already memorized training data. Combining theoretical analysis based on neural networks with random features and systematic empirical experiments, the study reveals for the first time that consistency distillation not only accelerates inference but also actively suppresses unstable feature directions associated with memorization. The results demonstrate that this approach substantially reduces memory transfer while preserving or even enhancing sample generation quality, thereby effectively improving the trade-off between memorization and generalization.