SoftNav: Injecting 3D Scene Tokens into VLMs for Embodied Navigation

2026-07-16

AI Summary

This work addresses the insufficient synergy between 3D scene understanding and navigation reasoning in goal-oriented embodied navigation, as well as the representational gap introduced by textual mediation of 3D information. The authors propose representing detected objects or frontier regions as continuous 3D soft tokens, which are directly injected into the hidden layers of a frozen vision-language model (VLM) via a lightweight projector. This approach achieves, for the first time, entity-level fusion of 3D scene tokens without requiring textual serialization, while keeping both the pretrained VLM and the 3D encoder frozen. The method significantly improves sample efficiency and generalization, attaining success rates of 74.2%, 68.3%, and 66.7% across three splits on HM3D-OVON—outperforming existing approaches—and demonstrates zero-shot transferability to GOAT-Bench, SG3D, and real-world robotic platforms.

Abstract

In goal-directed embodied navigation, where an agent must locate a specified target in an unseen environment, 3D scene understanding and navigation reasoning must work in concert. Current approaches transmit 3D scene information to vision-language models (VLMs) through text, suggesting a representation gap in our tested configurations; a controlled ablation confirms that direct embedding-level transfer significantly outperforms the evaluated text serialization formats. We introduce SoftNav, which injects entity-level 3D continuous representations -- one token per detected object or frontier -- into a VLM's hidden space as soft tokens through a lightweight projector. With the 3D encoder and VLM frozen, only ~1,200 samples and ~17M trainable parameters are needed. On HM3D-OVON, SoftNav achieves 74.2%/68.3%/66.7% SR across three splits, surpassing all prior methods in both SR and SPL; the same navigation policy transfers zero-shot to GOAT-Bench (67.2% SR), SG3D (47.2% s-SR), and real-world robot deployment without retraining or architectural modification. Injecting 3D scene tokens directly into VLMs bridges the representation gap, enabling transferable navigation with minimal training.

Problem

• embodied navigation
• 3D scene understanding
• vision-language models
• representation gap
• goal-directed navigation

Innovation

• 3D scene tokens
• vision-language models
• embodied navigation
• soft token injection
• zero-shot transfer