Anticipate Before Acting: Future-State-Conditioned Vision-Language Navigation

2026-07-20

AI Summary

This work addresses the limitation of existing end-to-end vision-and-language navigation (VLN) methods, which supervise only the current action and lack explicit modeling of future states. To overcome this, the authors propose the FSC-VLN framework, which introduces a future state conditioning mechanism: during training, future visual embeddings serve as supervisory signals to guide the policy toward learning forward-looking state representations, while requiring no future images at inference time. Built upon a causal vision-language model, FSC-VLN features a dual-branch architecture with separate future-query and action-query streams, employs a frozen visual encoder, and aligns future latent states through a dedicated target branch. The approach achieves significant improvements on the R2R val-unseen split across success rate (SR), oracle success rate (OSR), and success weighted by path length (SPL), with particularly strong performance on long-horizon trajectories.

Abstract

End-to-end vision-language navigation (VLN) with causal vision-language models can map instructions and egocentric observations directly to actions, but standard behavior cloning supervises only the next action and does not explicitly train the policy state to be predictive of future visual outcomes. We first ask a diagnostic question: if the policy is given an expert-trajectory future image as privileged input at training and testing time, is that additional visual evidence useful for choosing the current action? (These expert-trajectory future images are unavailable at test time in real deployment, so we use this setting only as a privileged-input diagnostic.) The answer is yes; this sanity check shows that future observations can provide rich, actionable cues. We then ask a deployable question: without accessing future images at inference, can we still benefit from future information by using a compressed future visual latent only as training supervision? We propose Future-State-Conditioned VLN (FSC-VLN), which adds a future-query token and aligns its hidden state to a frozen visual embedding $Δ$ steps ahead via a training-only target branch that is removed after training. On R2R val-unseen, FSC-VLN improves SR/OSR/SPL over a StreamVLN-style baseline under two training-data regimes, with larger gains on long-horizon episodes; ablations further support the dual-query design (separating future and action queries).

Problem

• vision-language navigation
• future-state prediction
• behavior cloning
• actionable cues
• policy state

Innovation

• vision-language navigation
• future-state conditioning
• end-to-end learning
• behavior cloning
• latent alignment