Limits of Confidence in Diffusion

2026-09-17

AI Summary

研究解决了离散扩散模型在生成序列时因令牌间依赖性导致的分布匹配问题，通过分析条件独立性和位置分布的关系，并在ScanAndAdd任务上验证了方法的有效性。

Abstract

Discrete diffusion, including remasking and uniform-state samplers, generate a sequence by writing multiple token positions per step, drawing each from a per-position distribution and choosing which positions to write from those same distributions. For domains of general interest (pixels, phonemes, or words) there are inherent dependencies between tokens. We show that a step matches the training distribution only when the positions it writes are conditionally independent given the tokens already fixed, that no product of per-position distributions can match a dependent group, and that per-position distributions do not determine whether a group is dependent: two joint distributions can have identical per-position marginals while differing in which combinations of values occur. On ScanAndAdd, a synthetic task whose joint distribution is available in closed form, we verify that every group of two or more undetermined positions a confidence ranking writes is dependent, and measure the generated distribution to be $29\times$ the sampling-noise floor total variation while per-sample metrics are $1.0$.

Problem

• Discrete diffusion
• conditional independence
• per-position distributions
• dependent groups
• confidence ranking

Innovation

• discrete diffusion
• conditional independence
• per-position distributions
• dependent groups
• ScanAndAdd