Bridging the 2D-3D Gap: A Hierarchical Semantic-Geometric Map for Vision Language Navigation

2026-05-25

AI Summary

This work addresses the disconnect between semantic and geometric information in vision-and-language navigation—particularly acute in zero-shot settings—caused by models’ limited 3D spatial reasoning capabilities. To bridge this gap, the authors propose a Hierarchical Semantic-Geometric Map (HSGM), which encodes the 3D environment into a structured, multi-channel top-down representation comprising geometric, semantic, and decision layers. This framework enables, for the first time, effective alignment between 3D geometric cues and vision-language models, while mitigating progress forgetting and hallucination in long-horizon navigation through instruction decomposition. High-level semantic planning is performed by a vision-language model operating on the HSGM, whereas low-level collision-free motion is delegated to classical path-planning algorithms, thereby decoupling semantic reasoning from action execution. Evaluated under zero-shot conditions on the R2R-CE and RxR-CE benchmarks, the method achieves state-of-the-art performance, surpassing several supervised approaches.

Abstract

Vision-Language Navigation (VLN) enables embodied agents to reach target locations in unseen environments by following language instructions. Despite recent progress with vision-language models (VLMs), a critical semantic-geometric gap remains: while VLMs excel at language and 2D visual understanding, they struggle with 3D spatial reasoning and fail to capture the causal dynamics between actions and spatial transitions, resulting in unreliable navigation, particularly in zero-shot settings. To bridge this gap, we propose a Hierarchical Semantic-Geometric Map (HSGM) that transforms 3D geometric information into a structured representation compatible with VLMs, effectively linking them to the physical world. Specifically, HSGM is represented as a multi-channel top-down map organized into three levels: (1) geometric level that records navigable regions and obstacles, (2) semantic level that represents objects and their relations, and (3) decision level that supports high-level task reasoning and goal selection. During navigation, the VLM acts as a high-level semantic planner, interpreting the spatial layout encoded in the HSGM to select geometrically valid waypoints, while low-level, collision-free movements between waypoints are executed by a classical path-planning algorithm, fully decoupling semantic reasoning from action execution. Additionally, complex instructions are decomposed into subtasks to alleviate the problem of progress forgetting or hallucinating in long-horizon navigation. Extensive experiments on R2R-CE and RxR-CE benchmarks demonstrate that our zero-shot framework achieves state-of-the-art performance and even outperforms several supervised methods. Code is available at https://github.com/Teacher-Tom/HSGM_public.

Problem

• Vision-Language Navigation
• Semantic-Geometric Gap
• 3D Spatial Reasoning
• Zero-shot Navigation
• Embodied AI

Innovation

• Hierarchical Semantic-Geometric Map
• Vision-Language Navigation
• 3D Spatial Reasoning
• Zero-shot Navigation
• Semantic-Geometric Gap