Same Attention, Different Truths: Put Logit-Lens over Visual Attention to Detect and Mitigate LVLM Object Hallucination

2026-08-07

AI Summary

This work addresses the hallucination problem in large vision-language models (LVLMs), where models generate descriptions of objects absent from images. Contrary to the prevailing view that insufficient visual attention causes such errors, Logit Lens analysis reveals that both real and hallucinated objects receive strong attention; the critical distinction lies in whether visual features can be consistently decoded into the corresponding target tokens. Building on this insight, the authors propose Detect-Mitigate, a training-free framework that distinguishes two distinct hallucination mechanisms—visual uncertainty and contextual priors—and introduces tailored mitigation strategies: High-Attention Region Masking (HARM) for the former and Visual Evidence Enhanced Decoding (VEED) for the latter. Evaluated across multiple object hallucination benchmarks, the approach achieves state-of-the-art performance in both detecting and mitigating LVLM hallucinations.

Abstract

Large Vision-Language Models (LVLMs) often suffer from object hallucination, generating objects that are absent from the image. Prior work largely attributes this to insufficient visual attention. However, we find that both real and hallucinated objects receive equally strong visual attention in the model's mid-to-late layers, suggesting that the key issue may not be how much the model attends, but what it attends to and why. To this end, we decode the visual features of high-attention regions using Logit Lens, and observe that regions corresponding to real objects can be correctly decoded to the target object tokens, whereas those for hallucinated objects cannot. Building on this, we identify two hallucination mechanisms: (i) visual uncertainty, triggered by semantically similar or confusable regions; masking these regions eliminates the hallucination. (ii) contextual prior, triggered by strong co-occurrence priors; even when the initially attended region is masked, the hallucination persists and attention drifts to other regions. Based on these findings, we propose a simple yet effective training-free Detect-Mitigate framework comprising a Logit-Lens Consistency Check to detect hallucination and targeted remedies: High-Attention Regions Masking (HARM) for visual uncertainty hallucination, and Visual Evidence Enhanced Decoding (VEED) for contextual prior hallucination. Our approach achieves state-of-the-art results on multiple hallucination benchmarks. Code will be available.

Problem

• object hallucination
• Large Vision-Language Models
• visual attention
• logit lens
• hallucination detection

Innovation

• Logit Lens
• object hallucination
• visual attention
• contextual prior
• training-free mitigation