🤖 AI Summary
This work addresses the information representation mechanism of deep learning models—particularly diffusion models—by proposing “neural entropy,” a novel information-theoretic measure that quantifies the essential information reconstructed and compressed by a neural network during denoising, i.e., recovered from data corrupted by the forward diffusion process. Methodologically, it is the first to dynamically couple information entropy with the diffusion path, defining neural entropy as a distribution- and noise-schedule-dependent functional grounded in the data distribution and forward diffusion dynamics. The measure is rigorously analyzed theoretically and empirically evaluated on image generation tasks. Experiments across multiple standard diffusion models show that neural entropy is substantially lower than the total entropy of the data, revealing that these models achieve highly efficient, structured semantic encoding with minimal information overhead. This work provides a computationally tractable and interpretable theoretical framework for understanding information compression in generative models.
📝 Abstract
We explore the connection between deep learning and information theory through the paradigm of diffusion models. A diffusion model converts noise into structured data by reinstating, imperfectly, information that is erased when data was diffused to noise. This information is stored in a neural network during training. We quantify this information by introducing a measure called neural entropy, which is related to the total entropy produced by diffusion. Neural entropy is a function of not just the data distribution, but also the diffusive process itself. Measurements of neural entropy on a few simple image diffusion models reveal that they are extremely efficient at compressing large ensembles of structured data.