See2Think: Do Multimodal Models Really Use Intermediate Visual States?

2026-07-29

AI Summary

Current evaluation methods struggle to determine whether multimodal large language models genuinely rely on intermediate visual states for reasoning, due to narrow task coverage, the solvability of samples using text alone, and an exclusive focus on final answers. This work proposes the See2Think framework, which introduces the See2ThinkBench benchmark—spanning 12 task categories—and the Visual Action-of-Thought protocol, establishing the first fine-grained evaluation paradigm specifically targeting the utilization mechanisms of intermediate visual states. By integrating a multitask open-ended benchmark, a controlled reasoning environment, and interventions that track visual actions, the framework systematically reveals models’ true behavioral dependence on visual states. The study identifies faithful rendering as a critical bottleneck: when task-relevant visual feedback is disrupted, accuracy drops by over 10 percentage points, with performance highly sensitive to model architecture and environmental conditions.

Abstract

Multimodal large language models increasingly use sketches, annotations, tools, and intermediate images during reasoning, but it remains unclear whether they truly rely on these visual states. Existing benchmarks are limited both by task collections with narrow coverage or partially text-solvable samples and by evaluations that emphasize final answers without diagnosing how intermediate visual states are generated, rendered, and used. We introduce See2Think, a unified evaluation framework comprising See2ThinkBench and Visual Action-of-Thought (VAoT). See2ThinkBench contains 1,200 open-ended, visually dependent problems across 12 task categories spanning 2D structured, 3D scene, and real-world reasoning. VAoT records textual thoughts, visual actions, rendered states, and subsequent reasoning under four controlled inference settings. Evaluating representative proprietary and open-source multimodal models, we find that visual reasoning is strongly model- and environment-dependent, with no single setting consistently dominating across tasks. Process analysis further shows that models usually select relevant visual operations, while faithful rendering remains the clearest bottleneck and high feedback uptake does not necessarily translate into accuracy gains. Under task-relevant corrupted feedback, models exhibit behavioral dependence on visual states, with accuracy dropping by over 10 percentage points in controlled interventions.

Problem

• multimodal models
• intermediate visual states
• visual reasoning
• evaluation benchmark
• visual dependency

Innovation

• multimodal reasoning
• intermediate visual states
• evaluation framework
• visual action-of-thought
• See2ThinkBench