Does Uniform Discrete Diffusion Need Time?

2026-09-25

AI Summary

This study addresses the frequent redundancy of explicit time conditioning in unified discrete diffusion models for language tasks. Through a combination of theoretical analysis and empirical optimization, we demonstrate that optimal predictors under finite data regimes are largely insensitive to time, motivating a simplified architecture that eliminates temporal conditioning. This design is systematically validated via large-scale language model experiments. Our findings challenge the conventional paradigm of relying on time embeddings, revealing that temporal dependence is negligible across most diffusion trajectories. Experimental results show that models without time conditioning achieve performance comparable to, or even exceeding, their time-conditioned counterparts, confirming that explicit temporal information is not essential. This work establishes a new pathway toward constructing more efficient discrete diffusion models.

Abstract

Uniform discrete diffusion models (UDMs) commonly use explicit time conditioning, but we find that it can often be unnecessary in practice. In this paper, we first show that the population-optimal UDM predictor generally depends on time: time controls how much the model should trust the observed context. We then show that this dependence can become negligible in finite-data settings relevant to language. When a corrupted training sequence remains much closer to its original clean sequence than to competing training sequences, the empirical-optimal predictor is nearly insensitive to time over most of the diffusion trajectory, where the guarantee weakens toward the high-noise endpoint. Empirically, trained language UDMs exhibit limited time sensitivity over most of the trajectory, while time-agnostic predictors remain competitive with, and often outperform, time-conditioned models across datasets and training objectives. These results challenge the use of explicit time conditioning in UDMs: although the population optimum depends on time, explicitly conditioning on it may often be unnecessary in practice.

Problem

• Uniform Discrete Diffusion
• Time Conditioning
• Language Modeling
• Diffusion Models

Innovation

• Uniform Discrete Diffusion
• Time Conditioning
• Time-Agnostic Predictors
• Finite-Data Settings
• Language Modeling