Seeing What Is Actually There: PriVE-Bench and PriVE-Tools for Counterfactual Evaluation of Agentic Visual Evidence in VLMs

2026-07-14

AI Summary

This work addresses the tendency of vision-language models (VLMs) to rely on linguistic and categorical priors rather than actual visual content, particularly when confronted with counterfactual images that contradict common sense. To systematically evaluate how tool-generated visual evidence—such as bounding boxes, cropped regions, zoomed panels, and object contours—can mitigate such prior-induced biases, the authors introduce PriVE-Bench, a novel benchmark, along with the PriVE-Tools suite. The framework employs paired original–counterfactual images and multidimensional metrics, including accuracy and prior error rate, to distinguish visually grounded responses from those driven by priors. Experimental results reveal that while certain models benefit from localized visual evidence, most still struggle to fully overcome prior bias, demonstrating that visual tools alone are not a universal remedy.

Abstract

Vision-language models (VLMs) often answer visual questions using learned language and category priors rather than grounding their predictions in the image itself. Counterfactual images provide a natural diagnostic setting for this failure mode: when visible evidence contradicts what is usually true, a grounded model should answer from the pixels, while a prior-following model will produce a canonical but visually incorrect response. However, existing counterfactual benchmarks mainly ask whether such prior-following behavior exists. In this paper, we ask a further question motivated by the rise of tool-augmented and agentic vision systems: can additional visual evidence views help VLMs reason against their priors? We introduce PriVE-Bench, a Prior-vs-Visual Evidence Benchmark that uses paired original and counterfactual images to distinguish visually grounded answers from prior-consistent errors. We further introduce PriVE-Tools, a controlled agentic-vision-inspired extension that evaluates whether tool-derived visual evidence -- including bounding boxes, crops, zoom panels, and contours -- improves grounding under the same counterfactual conflicts. Across open- and closed-source VLMs, we compare raw, paired-image, and tool-conditioned inputs using accuracy, prior-following error rate, and other-response rate. Our results show that visual evidence tools can help in some settings, especially when models can use localized evidence effectively, but they are not a universal remedy: several models continue to follow language and category priors even when relevant visual evidence is explicitly provided.

Problem

• vision-language models
• counterfactual evaluation
• visual grounding
• prior bias
• agentic vision

Innovation

• counterfactual evaluation
• visual grounding
• vision-language models
• tool-augmented reasoning
• prior-vs-evidence