How Do VLMs Fail? Vision-Operation Misalignment in Compositional VQA

2026-07-17

AI Summary

This work investigates the failure mechanisms of vision-language models in compositional visual question answering, attributing errors to mismatches between visual grounding and required reasoning operations. The authors propose an operation-centered analytical framework that integrates three complementary causal intervention techniques across Transformer layers to decompose failures by reasoning operation type and internal computational pathways. They formally define four distinct failure modes with divergent underlying mechanisms, revealing unique associations between each mode, visual grounding strength, and answer correctness. Notably, these modes exhibit dissociated routing patterns within the model’s internal pathways. Further analysis identifies that grounding failures propagate exclusively through feedforward networks, reasoning failures rely on deep-layer attention mechanisms, and attribute extraction failures localize to feedforward computations at the answer token position—providing a principled foundation for improving model reliability.

Abstract

Compositional visual question answering requires Vision-Language Models (VLMs) to execute multiple reasoning operations like object selection, spatial relation resolution, and attribute verification. Despite strong aggregate performance, the mechanistic basis of VLM failures on this task remains underexplored. To address this gap, we analyze vision-operation misalignment in VLMs by examining how failures relate to specific reasoning operations and the internal computational pathways through which they arise and propagate. We introduce an Operation-centric mechanistic framework that decomposes VLM failures by both the reasoning operation where they originate and the internal computational pathway through which they propagate. Our analysis reveals four mechanistically distinct failure modes: grounding failure, reasoning failure, attribute extraction failure, and language prior dominance failure. Each characterized by a unique relationship between visual grounding strength and answer correctness. Through three complementary causal interventions applied across all transformer layers, we further demonstrate a pathway dissociation: grounding failures route exclusively through the feedforward network, reasoning failures route through late-layer attention, and attribute extraction failures localize to the answer-position feedforward computation. This dissociation demonstrates that different failure types require fundamentally different corrective strategies, providing a principled foundation for targeted improvements to VLM reliability in multimedia reasoning.

Problem

• Vision-Language Models
• Compositional VQA
• Failure Modes
• Reasoning Operations
• Vision-Operation Misalignment

Innovation

• vision-operation misalignment
• compositional VQA
• mechanistic analysis
• failure modes
• causal intervention