🤖 AI Summary
This study investigates the impact of optical character recognition (OCR) information injection on the performance of vision-language models (VLMs), focusing on three representative architectures: Qwen3-VL, Phi-4, and InternVL3.5. Through causal intervention, activation difference analysis, principal component analysis (PCA), and cross-dataset directional transfer experiments, the work reveals for the first time that OCR signals propagate via a low-dimensional shared pathway, with the first principal component (PC1) accounting for 72.9% of the variance and demonstrating strong cross-dataset generalization. Notably, ablating the OCR module in Qwen3-VL-4B improves counting task accuracy by up to 6.9 percentage points, suggesting that OCR can interfere with non-text visual reasoning. These findings highlight the dual-edged role of OCR in multimodal fusion—beneficial for text-related tasks yet potentially detrimental to broader visual understanding.
📝 Abstract
Vision-language models (VLMs) can read text from images, but where does this optical character recognition (OCR) information enter the language processing stream? We investigate the OCR routing mechanism across three architecture families (Qwen3-VL, Phi-4, InternVL3.5) using causal interventions. By computing activation differences between original images and text-inpainted versions, we identify architecture-specific OCR bottlenecks whose dominant location depends on the vision-language integration strategy: DeepStack models (Qwen) show peak sensitivity at mid-depth (about 50%) for scene text, while single-stage projection models (Phi-4, InternVL) peak at early layers (6-25%), though the exact layer of maximum effect varies across datasets. The OCR signal is remarkably low-dimensional: PC1 captures 72.9% of variance. Crucially, principal component analysis (PCA) directions learned on one dataset transfer to others, demonstrating shared text-processing pathways. Surprisingly, in models with modular OCR circuits (notably Qwen3-VL-4B), OCR removal can improve counting performance (up to +6.9 percentage points), suggesting OCR interferes with other visual processing in sufficiently modular architectures.