Do Vision-Language Models See or Guess? Measuring and Reducing Textual-Prior Reliance with a Phrasing-Controlled Benchmark

2026-06-09

AI Summary

This work addresses the tendency of vision-language models to rely on question wording and textual priors rather than actual image content, leading to visually unsupported answers. The authors introduce a benchmark dataset comprising 540 images across six reasoning categories, each paired with four lexically varied but semantically equivalent questions—enabling, for the first time, the isolation and quantification of textual prior dependence by holding images fixed while varying prompts. Through comprehensive diagnostics—including image-ablation studies, LLM-based difficulty scoring, human relabeling, and in-context example matching—they find that all 11 evaluated models exhibit substantial performance drops on the most challenging variants, with open-source models’ accuracy falling to 1–9%. Further experiments demonstrate that GRPO-based post-training effectively improves robustness across all variants and generalizes to out-of-distribution data.

Abstract

Vision-language models (VLMs) are increasingly deployed where answers must follow from what is in the image, yet they often answer from textual priors, the question's phrasing together with memorized world knowledge, rather than from the image itself, which inflates benchmark scores and yields confident but ungrounded answers. Existing benchmarks rarely isolate this behavior, since each image is usually paired with a single fixed question. To measure the reliance, we build a 540-image benchmark across six reasoning categories and generate four question variants over the same images, so that phrasing rather than image content is the controlled variable. The hardest variant is written directly from the image to minimize text leakage. We benchmark eleven VLMs spanning small open-weight models to large closed-source systems: every model degrades on the hardest variant, and open models fall furthest. Our central diagnostic is a no-image ablation, which collapses the open-weight models to their text-only floor (1 to 9 percent). Three further analyses, LLM-rated difficulty, low base-to-final textual similarity, and human re-annotation, corroborate genuine image-dependence. In-context exemplars that match how a variant was built recover the most accuracy, and GRPO post-training of a small VLM yields consistent gains across all four variants that transfer to a held-out out-of-distribution set. Textual-prior reliance is measurable and partly trainable away.

Problem

• vision-language models
• textual priors
• image grounding
• benchmarking
• reasoning

Innovation

• textual-prior reliance
• phrasing-controlled benchmark
• vision-language models
• no-image ablation
• GRPO post-training