AgenticNav: Zero-Shot Vision-and-Language Navigation as a Tool-Calling Harness

2026-06-09

AI Summary

Existing zero-shot vision-and-language navigation methods struggle to generalize efficiently in continuous environments due to reliance on predefined waypoint prediction, insufficient utilization of depth information, and redundant memory mechanisms. This work proposes AgenticNav, a lightweight framework that reframes navigation as an agent interface between large vision-language models and the environment. AgenticNav introduces three key innovations: pixel-level action selection, on-demand depth querying, and a selective recall mechanism grounded in compact trajectory maps. Evaluated on the R2R-CE benchmark, the method achieves state-of-the-art zero-shot performance and demonstrates markedly stronger zero-shot generalization capabilities in real-world scenarios.

Abstract

Zero-shot vision-and-language navigation in continuous environments (VLN-CE) has recently become feasible with large vision-language models (VLMs). However, existing methods typically rely on learned waypoint predictors to propose navigable actions. This severely limits the model's action space and fails to leverage depth inputs effectively. Moreover, memory is commonly handled by accumulating long textual or visual histories with substantial irrelevant context, or by retrieving cross-episode experiences, which weakens the zero-shot setting. In this paper, we rethink zero-shot VLN-CE as an agentic interface between the VLM and the environment, and present AgenticNav, a lightweight navigation harness that exposes action, depth, and memory as callable tools. Instead of choosing from predicted waypoints, the action tool allows the VLM to directly select a target pixel in RGB observations, converting it into executable motion. Depth is exposed through an on-demand pixel-depth tool, enabling the VLM to request precise metric distances only where they matter. For memory, AgenticNav provides a compact map image summarizing the historical trajectory, paired with a recall tool that allows the VLM to selectively revisit past visual observations without overwhelming the prompt context. On the R2R-CE benchmark, AgenticNav establishes new state-of-the-art (SOTA) performance among zero-shot methods given the same VLM backbone. Real-world validation further highlights its zero-shot generalization compared to prior methods. Ablations show that our action tool design outperforms traditional waypoint predictors, and that depth tool and agentic memory further contribute to navigation performance.

Problem

• vision-and-language navigation
• zero-shot learning
• continuous environments
• depth perception
• memory management

Innovation

• tool-calling
• zero-shot navigation
• vision-language models
• agentic memory
• pixel-level action