Denoising Lévy Probabilistic Models

2024-07-26

arXiv.org

AI Summary

To address mode collapse induced by Gaussian noise in diffusion models and their limited capability in modeling heavy-tailed data, class-imbalanced distributions, and outliers, this paper proposes the first lightweight α-stable noise DDPM framework. Unlike existing approaches (e.g., LIM) that rely on intricate SDE derivations, our method directly incorporates α-stable distributions into both forward and reverse processes of DDPM. We design a stable-distribution-specific reparameterization technique and a Monte Carlo gradient estimator, and introduce an efficient backward scheduling strategy. Crucially, our approach avoids the computationally demanding Lévy–Itô decomposition, achieving a favorable balance between theoretical simplicity and implementation efficiency. Experiments demonstrate significant improvements over baselines in tail coverage, robustness to class imbalance, and outlier handling. Moreover, at comparable generation quality, our method reduces the number of reverse steps by 30–50%, enabling faster inference and lower memory consumption.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Diffusion Models for Vision
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Exploring noise distributions beyond Gaussian in diffusion models remains an open challenge. While Gaussian-based models succeed within a unified SDE framework, recent studies suggest that heavy-tailed noise distributions, like $alpha$-stable distributions, may better handle mode collapse and effectively manage datasets exhibiting class imbalance, heavy tails, or prominent outliers. Recently, Yoon et al. (NeurIPS 2023), presented the L'evy-It^o model (LIM), directly extending the SDE-based framework to a class of heavy-tailed SDEs, where the injected noise followed an $alpha$-stable distribution, a rich class of heavy-tailed distributions. However, the LIM framework relies on highly involved mathematical techniques with limited flexibility, potentially hindering broader adoption and further development. In this study, instead of starting from the SDE formulation, we extend the denoising diffusion probabilistic model (DDPM) by replacing the Gaussian noise with $alpha$-stable noise. By using only elementary proof techniques, the proposed approach, Denoising L'evy Probabilistic Models (DLPM), boils down to vanilla DDPM with minor modifications. As opposed to the Gaussian case, DLPM and LIM yield different training algorithms and different backward processes, leading to distinct sampling algorithms. These fundamental differences translate favorably for DLPM as compared to LIM: our experiments show improvements in coverage of data distribution tails, better robustness to unbalanced datasets, and improved computation times requiring smaller number of backward steps.

Problem

• Exploring non-Gaussian noise in diffusion models
• Addressing mode collapse and class imbalance
• Improving robustness and computation efficiency

Innovation

• Denoising Levy Probabilistic Models
• Replaces Gaussian with α-stable noise
• Improves data distribution coverage