On learning higher-order cumulants in diffusion models

2024-10-28

Machine Learning: Science and Technology

AI Summary

Diffusion models are commonly assumed to rely solely on Gaussian priors, raising fundamental questions about whether higher-order cumulants (i.e., n-point correlation functions) beyond Gaussian statistics are erased during the forward process and whether they can be recovered in the reverse process. Method: The authors combine cumulant generating functional analysis, score estimation theory, analytic derivation, and exactly solvable toy models, complemented by numerical validation in scalar lattice field theory. Contribution/Results: For variance-increasing forward processes, the authors provide the first rigorous proof that all higher-order cumulants are conserved; their information is fully encoded in the score function and precisely reconstructed by the reverse sampling process. Crucially, even when the forward process endpoint is approximately Gaussian, higher-order cumulants remain non-vanishing and are accurately recovered—challenging the conventional view that diffusion models fundamentally depend only on Gaussian assumptions. This establishes a theoretical foundation for modeling non-Gaussian structure in diffusion-based generative modeling.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Models for Web evolution

Abstract

To analyse how diffusion models learn correlations beyond Gaussian ones, we study the behaviour of higher-order cumulants, or connected n-point functions, under both the forward and backward process. We derive explicit expressions for the moment- and cumulant-generating functionals, in terms of the distribution of the initial data and properties of forward process. It is shown analytically that during the forward process higher-order cumulants are conserved in models without a drift, such as the variance-expanding scheme, and that therefore the endpoint of the forward process maintains nontrivial correlations. We demonstrate that since these correlations are encoded in the score function, higher-order cumulants are learnt in the backward process, also when starting from a normal prior. We confirm our analytical results in an exactly solvable toy model with nonzero cumulants and in scalar lattice field theory.

Problem

• Analyzing diffusion models' learning of non-Gaussian correlations
• Deriving expressions for cumulant-generating functionals in diffusion
• Demonstrating higher-order cumulant learning in backward process

Innovation

• Study higher-order cumulants in diffusion models
• Derive moment- and cumulant-generating functionals
• Learn correlations via score function in backward process