Detect Before You Leap: Mirage Detection in Vision-Language Models

2026-05-29

AI Summary

This work addresses the critical issue of high-confidence "hallucination" errors in vision-language models (VLMs) when visual evidence is absent—a particularly severe risk in medical and document-based visual question answering (VQA). The authors propose TC-LIA, a pre-response hallucination detection method that introduces a novel text-conditioned hierarchical internal alignment mechanism. By analyzing the alignment trajectories between image patches and question text across layers of CLIP ViT-H/14, TC-LIA integrates multi-dimensional features—including cosine similarity, top-k alignment, inter-layer gain, pixel statistics, zero-shot domain routing, and structured self-assessment—to enable model-agnostic, high-precision hallucination detection. Evaluated across five VQA domains, three input conditions, and twelve VLM backbones, TC-LIA achieves detection accuracies of 94.6–94.7% with hallucination rates below 3%, substantially outperforming baseline methods (21.7–66.6%).

Abstract

Vision-language models (VLMs) can produce confident visual answers even when the required visual evidence is missing, blank, or unrelated to the question. This failure mode, known as mirage (Asadi et al. 2026), is especially concerning in medical and document visual question answering, where plausible but visually ungrounded responses may be mistaken for image-based evidence. We study pre-release mirage detection: given an image-question pair, the goal is to determine whether a VLM should answer or abstain before producing a response. We propose Text-Conditioned Layer-wise Internal Alignment (TC-LIA), a model-agnostic method that probes patch-token representations across the layers of a CLIP ViT-H/14 vision encoder. TC-LIA projects layer-wise image patch tokens into the final CLIP embedding space and measures their similarity to the question embedding, allowing the method to track whether question-relevant visual evidence emerges across vision layers. The resulting alignment trajectory is summarized using final image-text cosine similarity, late-layer top-k patch-text alignment, early-to-late gain, and layer-wise slope. These features are combined with pixel-statistic blank/noise detection, zero-shot domain routing, and structured VLM self-assessment in an ensemble. Across five VQA domains, three input conditions, and twelve VLM backbones, the best systems achieve approximately 94.6-94.7% three-class detection accuracy with mirage rates below 3%, while baseline mirage rates range from 21.7% to 66.6%.

Problem

• mirage detection
• vision-language models
• visual question answering
• evidence grounding
• model abstention

Innovation

• mirage detection
• vision-language models
• layer-wise alignment
• model-agnostic probing
• visual question answering