BlockGen: Flexible Blockwise Sequence Modeling with Hybrid Samplers

2026-06-01

AI Summary

This work addresses the lack of systematic comparison between uniform and mask-based diffusion strategies in discrete diffusion models for block-wise sequence generation, as well as the limited efficacy of existing correction mechanisms in accurately repairing tokens with high error probability. To this end, we propose BlockGen, a block-level sequence generation framework that integrates autoregressive (AR) modeling with diffusion processes, enabling fine-grained control over multi-scale block training. We introduce an AR-guided predict-and-correct sampling strategy (ARPC) that regenerates low-confidence tokens without requiring an external verifier. Experimental results on GSM8K and OpenWebText demonstrate that mask diffusion slightly outperforms uniform diffusion under large block sizes (e.g., 16) and high numbers of function evaluations (NFE), particularly in terms of accuracy and perplexity.

Abstract

Is the uniform-state diffusion framework a more powerful paradigm for discrete diffusion? Recent studies indicate that this may be the case. In combination with predictor-corrector samplers, uniform-state diffusion models (USDMs) produce samples of higher-quality than masked diffusion models (MDMs), and USDMs equal or outperform MDMs in downstream tasks, even though they exhibit greater perplexity. Two issues remain unresolved. First, existing work compares uniform and masked diffusion with un-informed correctors that re-inject noise at random positions, rather than targeting tokens most likely to be wrong. Second, prior work compares full-sequence diffusion models, so we do not know whether the same conclusion holds when tokens are generated block by block. To address these issues, we introduce BlockGen, a blockwise sequence model that we instantiate with both masked and uniform diffusion. BlockGen trains on a mixture of block sizes and its likelihood interpolates between AR and pure diffusion more finely than models with a fixed block size. BlockGen enables AR-informed predictor-corrector sampling (ARPC), which combines AR and diffusion predictions to re-generate unlikely tokens without an auxiliary verifier. Under ancestral sampling, uniform outperforms masked in the block-by-block setting, especially in the few-step regime. Under ARPC, the gap closes and reverses at high NFE. With block size $16$ on GSM8K, MDMs reach slightly higher accuracy than USDMs, and we observe a similar trend in Generative Perplexity on OpenWebText. Find our code at https://github.com/jdeschena/blockgen.

Problem

• uniform-state diffusion
• masked diffusion
• blockwise sequence modeling
• predictor-corrector sampling
• discrete diffusion

Innovation

• blockwise sequence modeling
• uniform-state diffusion
• AR-informed sampling
• predictor-corrector sampler
• hybrid block sizes