Adaptive Multi-Step Lookahead Decoding for Diffusion Language Models

2026-07-17

AI Summary

Existing diffusion language models typically employ fixed-depth, single-step look-ahead decoding strategies, which struggle to balance efficiency and accuracy in long-horizon generation and fail to accommodate the heterogeneity of intermediate states. This work proposes AdaLook, a novel framework that introduces, for the first time, a dynamic multi-step look-ahead mechanism guided by the variance of candidate scores. AdaLook adaptively decides whether to further unfold or expand search branches, thereby avoiding unnecessary deep computations and enabling re-initiation of look-ahead from informative intermediate states. By integrating masked diffusion language modeling with adaptive decision-making and branch expansion strategies, AdaLook substantially outperforms existing single-step approaches across multiple benchmarks, achieving comparable generation quality with significantly fewer decoding steps.

Abstract

Masked diffusion language models (DLMs) enable parallel text generation by iteratively refining masked tokens, offering a promising alternative to autoregressive decoding. Recent lookahead-based decoding methods improve the accuracy--efficiency trade-off by exploring future decoding states before committing token updates. However, existing approaches mainly rely on shallow one-step lookahead, which optimizes immediate information gain but can be suboptimal for longer-horizon decoding trajectories. Meanwhile, we find that a naive extension for deeper lookahead is also ineffective, as fixed-depth rollout introduces additional computation and cannot adapt to heterogeneous intermediate decoding states. Thus, in this work, we propose AdaLook, an adaptive lookahead framework for DLM decoding. AdaLook dynamically determines whether to continue rollout based on candidate-score variance and further enables branch expansion when intermediate rollout states require additional exploration. This design avoids unnecessary deep rollout while allowing the decoder to re-trigger lookahead from informative intermediate states. Experiments on various benchmarks and models demonstrate that AdaLook achieves a better accuracy--decoding steps trade-off than existing one-step lookahead decoding methods.

Problem

• diffusion language models
• lookahead decoding
• adaptive decoding
• parallel text generation
• decoding efficiency

Innovation

• adaptive lookahead
• diffusion language models
• multi-step decoding
• parallel text generation
• dynamic rollout