Visual Instruction Tuning Aligns Modalities through Abstraction

2026-06-02

AI Summary

This work investigates how visual instruction tuning integrates image information into the hierarchical architecture of large language models—a mechanism not well understood in prior studies. Through a systematic analysis employing probing, causal intervention, representational geometry comparison, and selective layer fine-tuning, the study reveals that visual features are predominantly embedded in the model’s intermediate semantic layers in a localized manner. Building on this insight, the authors demonstrate that fine-tuning only these intermediate layers achieves performance comparable to full-model fine-tuning across multiple vision-centric benchmarks, while substantially reducing training costs. These findings underscore the pivotal role of intermediate layers in cross-modal alignment and offer an efficient strategy for multimodal adaptation.

Abstract

Visual instruction tuning effectively adapts a pre-trained Large Language Model (LLM) to process image information alongside text. Yet, it remains unclear how visual features are embedded into the layer-wise hierarchy of abstractions of the LLM backbone. Across a diverse set of vision-language architectures, we show that instruction tuning primarily serves as a bridge, embedding visual features directly into the intermediate semantic layers of the LLM, bypassing the early layers devoted to unimodal processing. With probing analyses and causal interventions, we show that these intermediate layers are the semantic core of vision-language processing and play a critical role in the performance on a broad set of multimodal benchmarks. In addition, by comparing the geometry of semantically equivalent visual and textual representations, we find that fine-tuning extends and strengthens the existing abstraction phase, aligning visual features with pre-existing textual ones. Finally, we confirm the functional role of this localized alignment by restricting fine-tuning to intermediate layers alone: this strategy preserves the performance of full fine-tuning on vision-centric benchmarks while reducing training time. Our results suggest that multimodal integration is a localized phenomenon driven by the repurposing of the internal abstraction engine of the LLM.

Problem

• visual instruction tuning
• multimodal alignment
• large language models
• modality integration
• semantic abstraction

Innovation

• visual instruction tuning
• modality alignment
• intermediate semantic layers
• abstraction hierarchy
• multimodal integration