MissingBench-Verified: Probing Vision-Language Models' Inability to Detect Missing Object Parts

2026-07-20

AI Summary

This study addresses a critical yet overlooked failure mode in visual-language models: their poor performance in recognizing when key components are missing from objects in images, a task heavily confounded by prior biases and data scarcity. The authors introduce MissingBench-Verified, the first benchmark specifically designed to evaluate this capability, and systematically assess ten state-of-the-art models. They further investigate the efficacy of various mitigation strategies, including tool-assisted preprocessing (e.g., image cropping and contrast adjustment), self-directed reasoning, extended inference time, and fine-tuning. Results reveal consistently high failure rates across all models, with existing interventions proving largely ineffective. These findings highlight a fundamental limitation—models struggle to correct internal misjudgments even when provided with external evidence—underscoring the necessity for architectural or paradigm-level innovations in training and design.

Abstract

Vision Language Models (VLMs) are well known for hallucinating non-existent objects in images. Objects with missing parts present a unique challenge for VLMs, stemming from both real-world knowledge bias and the scarcity of such images in training data. We present MissingBench-Verified, a benchmark designed to evaluate a specific and practically relevant scenario: when vision-language models fail to recognize that an essential component of an object has been removed. Across ten leading models, we observe consistent and significant failure rates that persist even when external tool evidence explicitly contradicts the model's visual perception. We further ask whether granting models access to image processing tools (e.g., cropping, contrast adjustment) enables autonomous inspection to resolve these failures. We find that existing mitigation strategies, including tool-assisted verification, autonomous visual reasoning, longer reasoning durations, and fine-tuning on an easier dataset, provide negligible improvement, indicating that this failure mode cannot be addressed through current prompting or post-hoc correction techniques. Our findings highlight a fundamental limitation of current VLM for inspection and monitoring tasks and underscore the need for architectural or training-level interventions that enable models to override internal expectations when confronted with contradictory evidence.

Problem

• vision-language models
• missing object parts
• hallucination
• visual perception failure
• benchmark evaluation

Innovation

• vision-language models
• missing object parts
• hallucination
• benchmark evaluation
• tool-assisted verification