Unveiling the Potential of Diffusion Large Language Model in Controllable Generation

2025-07-06

AI Summary

Existing diffusion-based large language models (dLLMs) face three key challenges in controllable text generation: sequence-length sensitivity in the denoising process, high hallucination rates, and substantial inference overhead. This paper introduces Self-adaptive Schema Scaffolding (S³), the first framework to explicitly reveal and harness the structural modeling capability of dLLMs’ bidirectional attention mechanisms. S³ achieves fine-grained, structured output control via schema-guided injection, adaptive inference optimization, and context-aware structural modeling—without modifying the model architecture. It dynamically aligns generated sequences with target schemas during inference, significantly improving semantic consistency and structural adherence. Experiments demonstrate that S³ boosts structural fidelity by 65%, enhances content fidelity by 48%, reduces hallucination rate by 17%, and concurrently improves inference efficiency. This work establishes the first lightweight, general-purpose, and efficient technical pathway for controllable diffusion-based text generation.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Diffusion models, originally developed for image generation, have emerged as a promising alternative to autoregressive large language models (LLMs). We present a theoretical analysis comparing autoregressive and masked diffusion LLMs, revealing that the intrinsic bidirectional attention mechanism of diffusion LLMs (dLLMs) enables superior context modeling and generation controllability. However, existing dLLM applications face significant challenges in controllable generation: the native multi-step denoising process exhibits high sensitivity to sequence length, elevated hallucination rates, and prohibitive inference costs without specialized optimizations. To address these limitations, we propose extbf{S}elf-adaptive extbf{S}chema extbf{S}caffolding ($S^3$), a novel framework that enables dLLMs to generate structured outputs (e.g., JSON) while maintaining semantic fidelity and accelerating inference. Our approach injects the target schema structure into the output context, reducing unnecessary computation while improving controllability. Extensive experiments demonstrate that $S^3$ achieves substantial improvements: 65% increase in structural adherence, 48% enhancement in content fidelity, and 17% reduction in hallucination rates compared to baseline. These results establish both theoretical foundations and practical pathways for deploying diffusion models in controllable text generation tasks. Code and data will be publicly released.

Problem

• Enhancing controllability in diffusion large language models
• Reducing hallucination rates in text generation
• Optimizing inference costs for structured outputs

Innovation

• Diffusion LLMs enable superior context modeling
• Self-adaptive Schema Scaffolding improves controllability
• Schema injection reduces computation and hallucinations