SpatialCLI: Learning to Reason With Spatial Tools, Then Without Them

2026-07-30

AI Summary

This work addresses the capability gap in general vision-language models between high-level task reasoning and fine-grained visual perception. To bridge this divide, the authors propose a three-stage internalization framework: first, enhancing perception by integrating composable spatial tools—such as localization, segmentation, depth estimation, and pose prediction; second, refining tool usage through Cold-Start supervised fine-tuning followed by agent-based reinforcement learning; and third, internalizing tool-derived capabilities into the model itself via trajectory verbalization. The newly introduced SpatialCLI-Bench benchmark comprehensively evaluates compositional perception skills. Experiments demonstrate that the proposed approach boosts the success rate of Qwen3-VL-8B-Instruct on MindCube from 29.3% to 84.6% when using tools, and maintains a strong 73.8% even after internalization without external tools, substantially outperforming GPT-5.6 Sol.

Abstract

Vision-language models (VLMs) are increasingly used in embodied agents to interpret visual inputs, reason about spatial relationships, and make task-level decisions based on that reasoning. However, a fundamental capability mismatch remains: general VLMs can reason about the overall task but often miss the visual details that determine success, while specialist vision models can capture those details but cannot translate them into task-level decisions. In this work, we propose SpatialCLI, a framework that teaches VLMs to reason with spatial tools and progressively internalize the specialist perceptual capabilities they provide. SpatialCLI proceeds in three stages: (1) Call exposes specialist vision models as spatial tools to augment the VLM's perception; (2) Learn uses Cold-Start SFT and agentic RL to improve tool use; and (3) Internalize verbalizes successful tool-use trajectories to internalize specialist perceptual capabilities. We further introduce SpatialCLI-Bench, a 516-example benchmark for compositional perception across localization, segmentation, depth, and pose. On MindCube, SpatialCLI raises Qwen3-VL-8B-Instruct from 29.3% to 84.6% with tools, surpassing GPT-5.6 Sol with tools (72.1%), while retaining 73.8% without tools after internalization.

Problem

• vision-language models
• spatial reasoning
• embodied agents
• perceptual capability
• tool use

Innovation

• SpatialCLI
• vision-language models
• spatial reasoning
• tool internalization
• embodied AI