PreDiff-LM: Pretrained Discrete Masked Diffusion Language Modeling with Hybrid Attention

2026-07-27

AI Summary

This work addresses the misalignment between autoregressive pretraining and bidirectional denoising objectives by introducing a hybrid attention mechanism that preserves causal attention in prompt regions while enabling full bidirectional attention in masked regions, thereby effectively reconciling pretrained weights with diffusion modeling requirements. Built upon the GPT-2 Medium architecture and integrating discrete masked diffusion, the DiffuGPT objective, and pretrained initialization, the proposed method achieves an unconditional perplexity of 26.9 on WikiText-103—down from 34.1—and attains a MAUVE score of 0.78. It substantially improves text diversity, distributional quality, performance across four zero-shot tasks, and human preference ratings, all while requiring significantly fewer training steps, offering a novel paradigm for repurposing autoregressive models.

Abstract

Discrete masked diffusion language models support bidirectional generation and infilling, but adapting pretrained autoregressive (AR) transformers requires reconciling causal pretraining with bidirectional denoising. We study this problem at the level of attention rather than claiming AR-weight reuse itself as novel. PreDiff-LM preserves causal attention within the observed prompt while allowing full bidirectional attention within the masked target. Under a matched GPT-2 Medium, WikiText-103, 90K-step setup, this hybrid mask improves unconditional perplexity from 34.1 to 28.7 and MAUVE from 0.71 to 0.78 over uniform bidirectional attention with the same AR initialization. Attention adaptation also composes with a DiffuGPT-style objective adaptation, reaching 26.9 perplexity. Pretrained initialization reduces the steps required to reach perplexity below 50 from about 350K to 8K, although a compute-matched fine-tuned AR model remains stronger at equal scale (18.9 versus 28.7). Beyond perplexity, PreDiff-LM improves repetition, distributional quality, four zero-shot downstream tasks, and human preference over prior diffusion baselines. The results position hybrid attention as a complementary mechanism for adapting pretrained causal backbones, while making explicit the remaining quality and inference-efficiency gaps to optimized AR models.

Problem

• discrete masked diffusion
• autoregressive pretraining
• bidirectional denoising
• attention adaptation
• language modeling

Innovation

• hybrid attention
• masked diffusion language modeling
• pretrained adaptation
• bidirectional generation
• discrete diffusion