Self-Regulating Annealing in Heavy-Tailed Diffusion Models

2026-05-31

AI Summary

Standard Gaussian diffusion models struggle to effectively capture heavy-tailed data distributions, often yielding generated samples with insufficient tail fidelity. This work proposes a novel stochastic differential equation (SDE)-based sampler that, for the first time in heavy-tailed diffusion models (HTDMs), incorporates a state-dependent diffusion coefficient. This design naturally induces a self-regulating annealing mechanism that dynamically adjusts the effective noise scale to better align with the heavy-tailed characteristics of the data. Theoretical analysis and empirical experiments demonstrate that the proposed approach significantly enhances both tail fidelity and overall generation quality, thereby validating the necessity and efficacy of state-dependent mechanisms for modeling heavy-tailed distributions.

Abstract

Diffusion models have emerged as a leading framework for deep generative modeling. While the standard Gaussian formulation is theoretically convenient, its suitability for heavy-tailed datasets remains unclear. To address this, heavy-tailed diffusion models (HTDMs) extend the standard formulation by replacing the Gaussian distribution with a Student's t-distribution, thereby improving tail fidelity on heavy-tailed datasets. Although stochastic differential equation (SDE)-based sampling is possible in HTDMs, it has not been fully explored. In this paper, we propose an SDE-based sampler for HTDMs that explicitly incorporates a state-dependent diffusion coefficient. This state dependence naturally induces a self-regulating annealing mechanism by adaptively modulating the effective noise scale. We theoretically explore this mechanism and experimentally verify its necessity for reproducing samples from a heavy-tailed distribution.

Problem

• heavy-tailed diffusion models
• diffusion models
• Student's t-distribution
• SDE-based sampling
• tail fidelity

Innovation

• heavy-tailed diffusion models
• state-dependent diffusion coefficient
• self-regulating annealing
• SDE-based sampling
• tail fidelity