Reading Right, Answering Wrong: How Visual Configuration Changes Affect Evidence Use in VLMs

2026-09-22

AI Summary

研究探讨了视觉配置变化（如图像切片和标记排列）如何影响视觉-语言模型在回答问题时对证据的使用，并通过引导模型使用字段提示和自身转录来纠正大部分错误。

Abstract

Vision-language models (VLMs) have achieved strong performance on tasks such as visual question answering, yet small image resizes can turn correct answers into errors. We investigate whether changes in visual configuration, such as image tiling and token arrangement, contribute to this instability. Across seven checkpoints and four benchmarks, equally small resizes cause more correctness flips when they switch configurations. Surprisingly, in over half of these cases, models answer the question incorrectly but can still read the correct answer when told what to read. Furthermore, attention interventions in LLaVA-NeXT suggest that configuration changes can weaken the use of readable information during answering. We therefore guide models using field cues and their own transcriptions. With annotation assistance, these forms of guidance together correct 97.2% of errors with readable information. These findings show that configuration changes can affect how models use information they can still read.

Problem

• Visual Configuration
• Vision-Language Models
• Correctness Flips
• Attention Interventions
• Information Use

Innovation

• visual configuration changes
• image tiling and token arrangement
• attention interventions
• field cues and transcription guidance