Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models

2026-07-28

AI Summary

This work addresses the challenge that multimodal large language models (MLLMs) often fail to reliably leverage visual information when it conflicts with linguistic priors. The authors propose WhatIfVis, a diagnostic benchmark that evaluates model reliance on visual versus linguistic priors across five coarse-grained attributes through image reconstruction and context sensitivity analysis. Their findings reveal that while MLLMs retain visual information, their post-perceptual utilization remains poorly controlled. Supervised fine-tuning substantially enhances cross-domain controllability, and activation patching identifies specific network layers where trade-offs between vision and priors are mediated. Notably, the study demonstrates for the first time that learned vectors can dynamically modulate the vision–prior trade-off even without explicit instructions.

Abstract

Multimodal Large Language Models (MLLMs) achieve strong performance by integrating visual inputs with the rich priors of pretrained language models. However, they often fail on vision-centric tasks, especially when visual evidence conflicts with pretrained knowledge. We explore these failures separately using two diagnostic paradigms: (1) probing whether visual information is available, via image reconstruction, and (2) measuring multimodal context sensitivity, the extent to which the model follows visual context versus the language prior. To support the second, we introduce the WhatIfVis, a benchmark spanning five coarse-grained dimensions (spatial-temporal, color, count, size, and weight) whose questions admit answers from either the image or the prior. Our analysis yields three findings: (i) Coarse-grained visual evidence is preserved, as these attributes can be reconstructed from the final-layer image tokens of frozen MLLMs. Failures on questions about these attributes therefore point to post-perceptual utilization, rather than to degraded visual encoding during perception. (ii) Even when explicitly instructed to use or ignore visual evidence, vanilla models (without supervised fine-tuning on the WhatIfVis) show unstable visual context sensitivity. Supervised fine-tuning (SFT) improves this controllability and generalizes across domains, and activation patching further localizes the vision-versus-prior trade-off at architecture-specific depths across all six models. (iii) The vision-versus-prior trade-off is controllable along a learned vector. Applying this steering vector, even without any intent instruction, improves controllability over the vanilla model. Together, these results relocate the bottleneck, indicating that for the coarse attributes we study, MLLMs encode the visual evidence but cannot reliably control their reliance on it.

Problem

• Multimodal Large Language Models
• visual context sensitivity
• language prior
• vision-language conflict
• visual evidence utilization

Innovation

• multimodal large language models
• visual context sensitivity
• WhatIfVis benchmark
• activation patching
• steering vector