Feature Information Dynamics in Diffusion

2026-10-06

AI Summary

This study addresses the long-standing absence of quantitative theoretical foundations for the temporal dynamics of feature generation in diffusion models, where existing understanding largely relies on qualitative heuristics. We construct an information-theoretic framework to characterize feature generation dynamics by leveraging the I-MMSE identity and chain-rule information decomposition to precisely quantify differences in feature information density and their emergence timing across pixel-space and latent-space representations. Our analysis confirms the spectral autoregressive phenomenon and reveals significant information density disparities between representations under conditions such as Canny edge guidance, demonstrating that ordered generation mechanisms facilitate model training. By shifting from qualitative intuition to rigorous quantitative analysis, this work establishes a theoretical foundation for understanding the representational nature of diffusion models.

Abstract

Diffusion models generate data through a continuum of denoising problems, and are widely observed to reveal coarse structure before fine detail. Yet, this intuition is mostly empirical and qualitative. We introduce feature information dynamics, an information-theoretic framework for localizing when a feature is generated during diffusion. Using the I-MMSE identity, we connect the rate of feature mutual information change to a gap between optimal unconditional and feature-conditional denoising losses, yielding practical estimators for feature information density. We further develop a chained decomposition that separates shared from incremental information in a feature hierarchy. We use this framework first to quantitatively confirm spectral autoregression in pixel diffusion, and then to extend the analysis beyond frequency: under a class $\to$ mask $\to$ Canny conditioning chain, the per-feature information densities differ across pixel, SDVAE, VAVAE, and RAE, exposing fundamental differences between these representations and suggesting that ordered generation could be beneficial for training diffusion models. Our code is available at https://github.com/AI4Science-WestlakeU/feature-information-dynamics.

Problem

• diffusion models
• feature information dynamics
• denoising
• information theory
• representation learning

Innovation

• Diffusion Models
• Feature Information Dynamics
• Information Theory
• I-MMSE Identity
• Spectral Autoregression