UNIFUSION: Adapting Autoregressive Language Models into Discrete Diffusion under a Unified Reverse-Rate Objective

2026-07-27

AI Summary

Existing methods struggle to effectively adapt pretrained autoregressive language models to uniform-noise diffusion processes due to misalignment between their objective functions and parameterizations. This work proposes UNIFUSION, a framework that establishes theoretical connections among diverse discrete diffusion models by introducing a unified reverse-rate objective based on generalized KL divergence. UNIFUSION further incorporates a universal x₀ interface that enables flexible switching between mask and uniform noise kernels. Coupled with a continuous pretraining strategy, it achieves the first direct transfer from GPT-2 checkpoints to uniform diffusion models. At 124M and 355M scales, UNIFUSION attains the best trade-off between generation perplexity and token entropy under 256-step sampling and sets new state-of-the-art results among diffusion models of comparable size on WinoGrande, SIQA, and BBH reasoning benchmarks.

Abstract

Existing methods mainly adapt pretrained autoregressive (AR) language models to masked diffusion, whereas we directly adapt them to uniform-noise diffusion, where every token remains editable during sampling. However, adapting AR checkpoints across corruption kernels remains challenging because existing DLMs use different objectives and prediction parameterizations. We establish connections among SEDD, MDLM/GIDD, M2S, and Neural CTMC by expressing their conditional losses as a single generalized Kullback--Leibler objective over model reverse rates. We further derive conversions from clean-token predictions to concrete-score, posterior-mean, and exit-rate/jump parameterizations, yielding a shared \(x_0\) interface that supports switching between mask and uniform kernels. Building on these connections, we propose \ours{}, a simple continual pre-training approach for directly adapting pretrained GPT2 checkpoints to uniform-noise diffusion. Through systematic evaluation of 124M- and 355M-parameter models, we show that \ours{} steadily improves the trade-off between generative perplexity (GenPPL) and unigram entropy as the sampling budget increases from 16 to 256 steps. At 256 steps, \ours{}-S and \ours{}-M achieve GenPPL/entropy pairs of \(97.783/5.2626\) and \(71.516/5.6669\), respectively; no evaluated model at the same scale simultaneously outperforms \ours{} on both metrics. At both scales, \ours{} also achieves the highest WinoGrande, SIQA, and BBH accuracy among the compared diffusion models.

Problem

• autoregressive language models
• uniform-noise diffusion
• diffusion language models
• reverse-rate objective
• corruption kernels

Innovation

• uniform-noise diffusion
• reverse-rate objective
• autoregressive adaptation
• discrete diffusion models
• unified parameterization