Hallucinations Leave a Grounding Signature:Verifier-Guided Decoding for Selective Object Correction

2026-07-30

AI Summary

This work addresses the prevalent issue of object hallucination in large vision-language models, where generated text often references entities absent from the input image. Existing mitigation strategies suffer from coarse-grained interventions due to insufficient diagnostic capability for fine-grained alignment errors, inadvertently compromising coverage of genuine objects. To overcome this, we propose Verifier-Guided Decoding (VGD), a novel framework that leverages the discovery that hallucinated objects leave identifiable intrinsic grounding signatures (IGS) within attention mechanisms. A lightweight verifier dynamically detects high-risk token generations during decoding and enables precise intervention through KV cache rollback, synonym suppression, and controlled regenerative refinement. Our method achieves state-of-the-art performance on CHAIR and AMBER-G benchmarks: it reduces the CHAIR score by 43.6% on AMBER-G while preserving 99.6% true object coverage, and lowers CHAIR_i and CHAIR_s by 37.0% and 30.4%, respectively, on CHAIR-MSCOCO without sacrificing caption length.

Abstract

Large vision-language models (LVLMs) often hallucinate objects that are absent from an image. Despite recent progress, existing mitigation methods still lack reliable object-level grounding diagnostics and therefore tend to apply coarse-grained interventions, which can impair visual understanding, shorten responses, and reduce coverage of genuinely grounded objects. The key challenge is thus to detect, during generation, whether each emerging object mention is supported by reliable visual evidence, so that hallucination can be mitigated selectively. Yet output confidence reflects next-token plausibility rather than visual support, allowing language priors to make absent objects appear certain. We show that the missing diagnostic evidence is encoded in an Intrinsic Grounding Signature (IGS), a distributed signed attention pattern that remains informative for such confident hallucinations. Based on IGS, we propose Verifier-Guided Decoding (VGD), a decoding framework in which a lightweight verifier examines each emerging object mention, rolls back the KV cache when the mention is identified as high risk, suppresses the object and its synonyms, and regenerates the affected continuation. Because VGD intervenes only on object mentions identified as high risk, it reduces object hallucination while preserving the model's original visual understanding and grounded object coverage. Experiments on CHAIR and AMBER-G show that VGD achieves state-of-the-art object hallucination reduction: at @rec90, it cuts AMBER-G CHAIR by 43.6\% while retaining 99.6\% of grounded-object coverage, and reduces CHAIR-MSCOCO CHAIR$_i$/CHAIR$_s$ by 37.0\%/30.4\% without shortening captions.

Problem

• object hallucination
• visual grounding
• vision-language models
• hallucination mitigation
• grounding diagnostics

Innovation

• Intrinsic Grounding Signature
• Verifier-Guided Decoding
• object hallucination
• vision-language models
• selective correction