Reading Images Like Texts: Sequential Image Understanding in Vision-Language Models

2025-09-23

AI Summary

Current vision-language models (VLMs) adopt a sequential image processing paradigm, fundamentally misaligning with the human visual system’s parallel dual-path architecture—distinguishing content recognition from spatial perception—and suffer from opaque, uninterpretable internal representations. Method: We propose a decoupling analytical framework that, for the first time, systematically uncovers a two-stage content understanding evolution in VLMs: attribute identification followed by semantic disambiguation; we further establish a geometric theory of positional representation. Leveraging this, we design an instruction-free token compression algorithm and a RoPE scaling technique. Contributions/Results: Through textual image token mapping, visualized decoder analysis, and empirical evaluation, we demonstrate significant improvements in decoding efficiency and spatial reasoning capability. This work establishes an interpretability foundation and actionable design principles for next-generation VLM architectures.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Vision-Language Models (VLMs) have demonstrated remarkable performance across a variety of real-world tasks. However, existing VLMs typically process visual information by serializing images, a method that diverges significantly from the parallel nature of human vision. Moreover, their opaque internal mechanisms hinder both deeper understanding and architectural innovation. Inspired by the dual-stream hypothesis of human vision, which distinguishes the"what"and"where"pathways, we deconstruct the visual processing in VLMs into object recognition and spatial perception for separate study. For object recognition, we convert images into text token maps and find that the model's perception of image content unfolds as a two-stage process from shallow to deep layers, beginning with attribute recognition and culminating in semantic disambiguation. For spatial perception, we theoretically derive and empirically verify the geometric structure underlying the positional representation in VLMs. Based on these findings, we introduce an instruction-agnostic token compression algorithm based on a plug-and-play visual decoder to improve decoding efficiency, and a RoPE scaling technique to enhance spatial reasoning. Through rigorous experiments, our work validates these analyses, offering a deeper understanding of VLM internals and providing clear principles for designing more capable future architectures.

Problem

• Analyzes object recognition and spatial perception in Vision-Language Models
• Identifies a two-stage process from attributes to semantic disambiguation
• Proposes methods to improve decoding efficiency and spatial reasoning

Innovation

• Deconstructs vision into object and spatial pathways
• Introduces token compression via plug-and-play decoder
• Enhances spatial reasoning with RoPE scaling technique