Path-Dependent Denoising: A Non-Conservative Field Perspective on Order Collapse in Diffusion Language Models

2026-05-10

AI Summary

Diffusion language models struggle to achieve truly order-agnostic generation in fast parallel decoding due to their sensitivity to denoising order. This work formalizes the resulting “order collapse” problem for the first time as an issue of compatibility among local conditional distributions. Adopting a non-conservative field perspective, it introduces order-induced pseudo-joint distributions and local denoising circulations to uncover the root cause of path dependence. Building upon probabilistic graphical models and circulation decomposition, the paper proposes a theoretical framework that enables diagnosis of order-freedom solely at inference time. This framework effectively disentangles path dependence from two distinct error sources: conditional dependency errors arising from parallel updates and order-specific estimation errors, thereby providing the first quantifiable analytical tool for evaluating the order-agnostic properties of diffusion language models.

Abstract

Diffusion language models (DLMs) offer a structural alternative to autoregressive generation: denoising can update tokens in arbitrary orders or in parallel rather than along a fixed left-to-right chain. In practice, fast DLM decoding remains strongly order-sensitive and often drifts toward autoregressive-like trajectories. We trace this tension to compatibility. At each reverse-time step, a DLM provides local denoising conditionals over the unresolved tokens. Arbitrary-order denoising becomes well defined when these local conditionals compose into order-invariant pseudo-joints. We formalize this view by defining order-induced pseudo-joints and a local denoising circulation: the log-ratio between the two pseudo-joints obtained by swapping a pair of unresolved positions. This circulation is zero under compatible conditionals, and global order gaps decompose into sums of local circulations along adjacent swaps. We further separate incompatibility-driven path dependence from conditional-dependence error in parallel updates and from order-specific estimation error. The resulting framework provides inference-only diagnostics for testing when DLM decoding is genuinely order-free.

Problem

• diffusion language models
• order collapse
• path dependence
• non-conservative field
• denoising compatibility

Innovation

• path-dependent denoising
• non-conservative field
• order-induced pseudo-joints
• local denoising circulation
• compatibility in diffusion models