OrientSAM: Mitigating Camera-Centric Shortcut in Multimodal Spatial Reasoning via Orientation-Aware Spatial Alignment

2026-07-20

AI Summary

This work addresses the tendency of multimodal large language models to over-rely on camera-centric perspectives in spatial reasoning, which limits their generalization to non-camera viewpoints such as object- or agent-centered frames. To mitigate this viewpoint shortcut, the authors propose OrientSAM, a novel framework that explicitly models object orientation as a key factor for improving perspective generalization. OrientSAM introduces orientation-aware tokens and a Fourier-based angular encoding mechanism, complemented by a curriculum learning strategy and an automated data construction pipeline to enhance spatial alignment and viewpoint robustness. Experiments demonstrate that OrientSAM significantly outperforms existing methods across multiple benchmarks—including Spatial-MM, ViewSpatial, and 3DSRBench—with particularly strong performance on tasks requiring non-camera perspectives, human-centric reasoning, and orientation sensitivity.

Abstract

Multimodal large language models (MLLMs) still struggle with spatial reasoning that requires perspective transformation. In particular, they often rely on camera-centric cues rather than reasoning from the reference object's viewpoint, leading to systematic errors in non-camera reference settings. In this paper, we first analyze this failure mode and show that object orientation is a key factor underlying such camera-centric shortcut behavior. To address this issue, we propose OrientSAM, an orientation-aware spatial alignment framework for multimodal models. OrientSAM injects explicit orientation information into multimodal representations through orientation-aware tokens and Fourier-based angle encoding, and further adopts a curriculum learning strategy to progressively improve perspective-aware reasoning. In addition, we build a spatial data construction pipeline to generate orientation-aware spatial supervision from large-scale images. Experiments on Spatial-MM, ViewSpatial, and 3DSRBench show that OrientSAM consistently outperforms strong baselines, especially on non-camera-view, person-centric, and orientation-sensitive tasks. The results further demonstrate that explicit orientation modeling is important for mitigating camera-centric shortcut behavior and enabling more robust allocentric spatial reasoning in multimodal models.

Problem

• spatial reasoning
• camera-centric shortcut
• perspective transformation
• orientation awareness
• multimodal models

Innovation

• orientation-aware
• spatial alignment
• camera-centric shortcut
• multimodal reasoning
• Fourier-based angle encoding