From Routes to Steps: Separating Semantic Progress from Local Execution in Vision-and-Language Navigation

2026-08-04

AI Summary

Existing vision-and-language navigation methods struggle to distinguish between semantic progress errors and local execution errors, leading to difficulties in recovering from trajectory deviations. This work proposes Route2Step, a novel framework that explicitly decouples semantic progress tracking from action generation, assigning these tasks to dedicated instruction analysis and action generation modules, respectively. It introduces E-SPA, an alignment mechanism that enables effective state supervision and error correction without requiring manual temporal annotations. Trained with only minimal direct action supervision and state-level correction samples, Route2Step achieves a success rate of 55.3% (+7.2%) and a path-length-weighted success rate of 48.2% (+4.9%) on R2R-CE, demonstrating its efficacy in both simulated and real-world indoor and outdoor environments.

Abstract

Vision-and-Language Navigation (VLN) requires an agent to follow a route-level instruction by executing its constituent steps from egocentric visual observations. Existing VLM-based navigators typically supervise both capabilities through next-action prediction alone, making progress-tracking errors difficult to distinguish from execution errors. When an agent deviates from the route, a corrective action label may recover the next movement but does not indicate whether the agent selected the wrong sub-instruction or failed to execute the correct one. Consequently, the agent may continue making decisions from an erroneous progress state. To resolve this ambiguity, we propose \textbf{Route2Step}, a framework that decouples semantic progress tracking from action generation through an explicit step-level interface. The Instruction Analysis Module ($\mathcal{M}_{\mathrm{IA}}$) predicts this state from the global instruction and visual history. Conditioned on the predicted state and recent observations, the Action Generation Module ($\mathcal{M}_{\mathrm{AG}}$) generates local action chunks. To supervise the progress state without manual temporal labels, E-SPA, a step-alignment procedure, associates sub-instructions with their corresponding portions of route-level demonstrations. These alignments enable state supervision for incorrect progress estimates, while direct action supervision is reserved for rollout groups that repeatedly fail under the correct active sub-instruction. On R2R-CE, Route2Step improves SR from 48.1\% to 55.3\% and SPL from 43.3\% to 48.2\%, using 190K state-level corrective samples while requiring only 11.5K directly action-supervised states. Experiments in real-world indoor and outdoor environments further demonstrate the practical applicability of Route2Step. The project page is: https://sisyphus-hxy.github.io/Route2Step/.

Problem

• Vision-and-Language Navigation
• semantic progress tracking
• action execution
• step-level alignment
• route deviation

Innovation

• Vision-and-Language Navigation
• Progress Tracking
• Action Generation
• Step-Level Alignment
• Decoupled Architecture