Spatial-IQ: Deconstructing Spatial Intelligence via Hierarchical Capability Tests

2026-07-24

AI Summary

This work addresses the poor performance of multimodal large language models (MLLMs) on spatial reasoning tasks and the lack of diagnostic tools to disentangle failures arising from perceptual versus cognitive deficits. The authors propose the first hierarchical diagnostic framework grounded in established stages of human spatial cognition, decomposing object counting in 3D stacked structures into nine perceptual and cognitive subtasks. Leveraging mental rotation as a probing mechanism, they programmatically generate approximately 80,000 samples using NVIDIA Isaac Sim for systematic evaluation. Models are trained using chain-of-thought supervision and reinforcement learning with verifiable rewards across three output modalities: free-form text, image-based multiple choice, and image editing. Experiments demonstrate that the framework not only exposes models’ reliance on shortcut behaviors but also serves as an effective training signal, substantially improving accuracy on the target task and enhancing cross-subtask spatial consistency.

Abstract

Multimodal large language models (MLLMs) excel at visual interpretation but fail on spatial reasoning tasks that humans solve reliably. Existing benchmarks evaluate these models as black boxes, limiting their ability to identify the underlying causes of lower performance: when a model fails a spatial reasoning task, it remains difficult to ascertain whether the hurdle is perceptual, such as recognizing object boundaries, or cognitive, such as reasoning about occlusion to infer hidden geometry. We introduce Spatial-IQ, a hierarchical diagnostic framework that decomposes object counting in stacked 3D structures into 9 perceptual and cognitive sub-tasks organized by the developmental stages of human spatial cognition, with mental rotation as an additional target probe. Using NVIDIA Isaac Sim, we procedurally generated a diverse dataset of roughly 80,000 stacked 3D structures with per-task ground truth. We evaluate models across three output formats (free-response text, multiple-choice images, and image editing) alongside a human baseline. The Spatial-IQ framework shows that top-performing models often succeed at the target task (object counting) without succeeding on the lower-level sub-tasks intended to support it, and that models differ in how much of these hierarchical chains they preserve, often revealing shortcut behavior that raw target-task accuracy alone would obscure. Finally, we demonstrate that training models with chain-of-thought (CoT) supervision over our hierarchical sub-tasks, combined with reinforcement learning with verifiable rewards, significantly improves both spatial consistency across sub-tasks and target-task accuracy, supporting the value of the proposed decomposition as both a diagnostic tool and a training signal.

Problem

• spatial reasoning
• multimodal large language models
• perceptual vs cognitive
• object counting
• hierarchical decomposition

Innovation

• Spatial-IQ
• hierarchical diagnostic framework
• spatial reasoning
• chain-of-thought supervision
• multimodal large language models