Before the Token Commits: Trajectory-Level Benchmarking of Visual Hallucinations in Diffusion VLMs

2026-09-28

AI Summary

This study addresses the limitation of existing benchmarks for diffusion-based vision-language models (VLMs), which evaluate only final outputs and cannot trace when hallucinations become established during generation. To overcome this, we introduce DynaHall, the first trajectory-level hallucination measurement protocol, which records intermediate unmasking states to reveal that visual hallucinations stabilize well before the commitment step. Furthermore, we propose Pre-commitment Gradient Search (PGS), a method that applies gradient-guided editing during early generation stages to intervene in error-prone trajectories. Experimental results demonstrate that PGS significantly reduces hallucination false-positive rates while preserving general model capabilities, achieving effective transferability across architectures. This work provides a novel trajectory-level perspective for understanding and mitigating hallucinations in diffusion VLMs.

Abstract

Multimodal diffusion language models generate responses by iteratively unmasking tokens, making each answer the endpoint of a multi-step trajectory rather than an immediate commitment. Hallucination benchmarks built for autoregressive models evaluate only the final output, and therefore cannot determine whether an unsupported claim in diffusion VLMs appears late or has already stabilized before any answer token is revealed. We introduce DynaHall, a trajectory-level benchmark of annotation-backed binary visual propositions covering object existence, counting, attributes, and relations, with controlled hard negatives graded by visual prior. DynaHall is paired with a commitment-aware protocol that records the intermediate answer tendency at every unmasking step alongside the committed output. Across five diffusion VLMs from three architecture families, visual hallucination is settled before commitment: an unsupported answer is already the preferred state while the answer position is still masked, and later unmasking steps rarely reverse it, so the failure is not introduced at the write step. This holds across decoding schedules, answer formats, and open-ended generation. DynaHall also exposes failures hidden by final-output metrics, including counting and relation collapse, prior-driven false positives, and attribute errors whose direction changes by type. Guided by this diagnosis, PGS (Pre-commitment Gradient Steering) edits still-masked answer states to reduce false positives, bringing the affirmation rate close to balance, and transfers to another architecture without degrading general ability. DynaHall and PGS suggest that hallucination should be measured and mitigated along the generation trajectory of diffusion VLMs, not only at the final answer.

Problem

• Visual Hallucination
• Diffusion VLMs
• Trajectory-Level Benchmarking
• Token Unmasking

Innovation

• Diffusion VLMs
• Visual Hallucination
• Trajectory-level Benchmarking
• Pre-commitment Gradient Steering
• Commitment-aware Protocol