Dynamic Infilling Anchors for Format-Constrained Generation in Diffusion Large Language Models

2026-06-03

AI Summary

Fixed anchor points in format-constrained text generation often lead to premature truncation or redundant content, making it difficult to simultaneously ensure structural correctness and semantic coherence. This work proposes Dynamic Insertion Anchors (DIA), a method that dynamically estimates anchor positions within diffusion-based large language models without requiring additional training, enabling flexible adjustment of output length through iterative filling. DIA introduces, for the first time, a training-free dynamic anchoring mechanism that overcomes the limitations of fixed-span constraints by integrating bidirectional attention with a parallel generation architecture, significantly enhancing both flexibility and accuracy. Extensive experiments on benchmarks such as GSM8K and MATH demonstrate that DIA substantially improves zero-shot format compliance and answer accuracy, confirming its effectiveness and robustness.

Abstract

Diffusion large language models (dLLMs) offer bidirectional attention and parallel generation, enabling them to exploit global context and naturally support format-constrained tasks like parseable JSON or reasoning templates. While straightforward fixed anchors can enforce such constraints, they often impose rigid spans, leading to truncated reasoning or redundant content. To overcome this, we propose Dynamic Infilling Anchors (DIA), a training-free method that dynamically estimates end-anchor positions to adjust generation length before iterative infilling. This flexible mechanism ensures structural correctness and semantic coherence, avoiding the inefficiencies of fixed-span methods. Experiments on reasoning benchmarks demonstrate that DIA substantially improves format compliance and answer accuracy, achieving significant zero-shot gains on GSM8K and MATH. These results establish DIA as a robust pathway toward reliable, structure-aware generation.

Problem

• format-constrained generation
• diffusion large language models
• fixed anchors
• structural correctness
• semantic coherence

Innovation

• Dynamic Infilling Anchors
• Diffusion LLMs
• Format-Constrained Generation
• Zero-shot Reasoning
• Structure-Aware Generation