Breaking the Factorization Barrier in Diffusion Language Models

2026-02-09

arXiv.org

AI Summary

This work addresses the limitation of conventional diffusion language models, which rely on fully factorized output distributions and struggle to effectively capture token dependencies during parallel generation, thereby forcing a trade-off between generation speed and textual coherence. To overcome this structural constraint, the authors propose the Coupled Discrete Diffusion (CoDD) framework—the first approach to break away from full factorization—by introducing a lightweight, tractable discrete probabilistic inference layer that enables efficient joint distribution modeling while maintaining compact parameterization. Integrated with a Transformer backbone, CoDD forms a hybrid diffusion architecture that significantly enhances generation quality across multiple model scales with minimal computational overhead, matches the reasoning performance of reinforcement learning baselines, avoids performance collapse in few-step generation, and substantially reduces latency.

Abstract

Diffusion language models theoretically allow for efficient parallel generation but are practically hindered by the"factorization barrier": the assumption that simultaneously predicted tokens are independent. This limitation forces a trade-off: models must either sacrifice speed by resolving dependencies sequentially or suffer from incoherence due to factorization. We argue that this barrier arises not from limited backbone expressivity, but from a structural misspecification: models are restricted to fully factorized outputs because explicitly parameterizing a joint distribution would require the Transformer to output a prohibitively large number of parameters. We propose Coupled Discrete Diffusion (CoDD), a hybrid framework that breaks this barrier by replacing the fully-factorized output distribution with a lightweight, tractable probabilistic inference layer. This formulation yields a distribution family that is significantly more expressive than standard factorized priors, enabling the modeling of complex joint dependencies, yet remains compact enough to avoid the prohibitive parameter explosion associated with full joint modeling. Empirically, CoDD seamlessly enhances diverse diffusion language model architectures with negligible overhead, matching the reasoning performance of computationally intensive Reinforcement Learning baselines at a fraction of the training cost. Furthermore, it prevents performance collapse in few-step generation, enabling high-quality outputs at significantly reduced latencies. Code available at: https://github.com/liuanji/CoDD

Problem

• diffusion language models
• factorization barrier
• parallel generation
• token dependencies
• joint distribution

Innovation

• Coupled Discrete Diffusion
• diffusion language models
• factorization barrier
• joint distribution modeling
• parallel generation