Problem
Research questions and friction points this paper is trying to address.
Diffusion Language Models
One-Step Block Generation
Noise-Conditioned Masked Denoiser
A Ticket from Marginals to Joints: Coupled-Noise Distillation for One-Step Block Generation in Diffusion Language Models
🤖 AI Summary
研究通过CONDOR方法，利用噪声条件教师和学生模型，实现扩散语言模型一步生成整块文本，提高了一步生成的连贯性和合法性。
📝 Abstract
Autoregressive language models commit one token per forward pass; diffusion language models commit a block of tokens over several steps. We ask whether a block can be committed in a single forward pass. We study this with a noise-conditioned masked denoiser: a data-independent Gaussian noise field is added to the mask embeddings so that, in principle, each sampled field selects one joint mode of the block. The established way of training such a model is to sample several fields per example and let them compete for the data, by winner-take-all or importance weighting. This gives the noise only coarse control: in our experiments, the information it carries grows roughly with the logarithm of the number of competing fields, and one-step outputs remain rarely coherent across the model sizes tested. We propose CONDOR (Coupled-Noise Distillation for One-Step Readout). A noise-conditioned teacher is trained with a random number of masked positions and winner-take-all. A student proposes a one-step block, retains selected tokens, and learns from the block obtained when the teacher refills the other positions in several steps under the same noise field; a noise-free masked-LM term on the ground truth anchors the student. Human evaluation on TinyStories shows a large gain in one-step legality while different noise fields still yield different blocks, at one forward pass per block.
Innovation
Methods, ideas, or system contributions that make the work stand out.
CONDOR
one-step block generation
noise-conditioned teacher
coupled-noise distillation
diffusion language models
🔎 Similar Papers
No similar papers found.
