SphMind: Towards Robust, Training-Free VLM-based Spatial Reasoning with a 360 Camera

2026-09-27

AI Summary

This work addresses the challenge of spatial reasoning in multimodal large models when processing 360° panoramic images, which suffer from geometric distortion and boundary discontinuity. We propose a training-free, plug-and-play framework that decouples semantic perception from geometric reasoning. Specifically, the framework incorporates spherical harmonic spatial graph modeling and equivariant transformation techniques, alongside a model-agnostic, inference-time geometric grounding mechanism with closed-loop optimization to achieve robust panoramic spatial understanding. Experimental results demonstrate that the proposed method yields an average improvement of over 21.4% on directional reasoning benchmarks and enhances rotation invariance by 5.9%, significantly outperforming existing baseline approaches.

Abstract

Omnidirectional or 360 cameras provide embodied AI agents with a holistic, wide field-of-view (FoV) view of their surroundings, motivating the use of Multi-modal Large Language Models (MLLMs) for omnidirectional spatial reasoning. However, most MLLMs are trained on conventional 2D perspective images and struggle with the severe distortions and wrap-around discontinuities induced by spherical geometry. Enabling them to generalize to non-Euclidean 3D spaces without retraining therefore remains challenging. We propose SphMind, a training-free, plug-and-play framework that decouples semantic perception from geometric reasoning. Rather than requiring MLLMs to learn spherical geometry internally, SphMind preserves their semantic capabilities while handling geometry externally. We introduce a Spherical Harmonics-based Spatial Graph (SHSG) that models spatial relationships through equivariant transformations on the sphere, together with Inference-Time Geometric Grounding (IGG), a model-agnostic closed-loop optimization process that aligns MLLM representations with spherical geometric constraints during inference. Experiments on three benchmarks show that SphMind achieves over 21.4% average improvement in directional reasoning on MP3D and Stanford2D-3D, outperforms prompt-engineering baselines by 8.7% on the real-world ODI-Bench, and improves rotational invariance by 5.9% under panorama rotations, without additional training or dataset-specific tuning. In-the-wild evaluations further show that SphMind resolves directional reasoning queries that baseline vision-language models fail to answer correctly.

Problem

• Omnidirectional spatial reasoning
• Multi-modal Large Language Models
• Spherical geometry distortion
• Training-free generalization
• 360 camera

Innovation

• Training-Free VLM
• Omnidirectional Spatial Reasoning
• Spherical Harmonics Spatial Graph
• Inference-Time Geometric Grounding
• Semantic-Geometric Decoupling