Generalized Discrete Diffusion from Snapshots

2026-03-22

AI Summary

This work addresses the limited flexibility and low training efficiency of discrete diffusion models in large-scale discrete state spaces by proposing a unified framework that accommodates arbitrary noise schedules. The approach introduces a snapshot-based latent variable formulation to construct the evidence lower bound (ELBO) for the reverse generative process, thereby unifying existing discrete diffusion models within a single theoretical framework. By integrating uniformization techniques, the method enables both flexible design and efficient training. Evaluated on large-vocabulary generation tasks, the proposed framework not only substantially outperforms current discrete diffusion approaches but also, for the first time, surpasses autoregressive models in both generation quality and training efficiency.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Search and Optimization: Mixed Discrete/Continuous Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

We introduce Generalized Discrete Diffusion from Snapshots (GDDS), a unified framework for discrete diffusion modeling that supports arbitrary noising processes over large discrete state spaces. Our formulation encompasses all existing discrete diffusion approaches, while allowing significantly greater flexibility in the choice of corruption dynamics. The forward noising process relies on uniformization and enables fast arbitrary corruption. For the reverse process, we derive a simple evidence lower bound (ELBO) based on snapshot latents, instead of the entire noising path, that allows efficient training of standard generative modeling architectures with clear probabilistic interpretation. Our experiments on large-vocabulary discrete generation tasks suggest that the proposed framework outperforms existing discrete diffusion methods in terms of training efficiency and generation quality, and beats autoregressive models for the first time at this scale. We provide the code along with a blog post on the project page : \href{https://oussamazekri.fr/gdds}{https://oussamazekri.fr/gdds}.

Problem

• discrete diffusion
• large discrete state spaces
• arbitrary noising processes
• generative modeling
• training efficiency

Innovation

• discrete diffusion
• snapshot latents
• uniformization
• evidence lower bound
• generative modeling