Token-Wise Latent Streaming from Slow Reasoners to Fast Planners for Dynamic Vision Language Navigation

2026-07-18

AI Summary

This work addresses the challenge in dynamic vision-and-language navigation (VLN) where slow language-based reasoning often leads to action lag and safety risks due to environmental changes. To overcome this, the authors propose SPARK-VLN, a dual-system framework that enables token-wise streaming of intermediate hidden states from a vision-language model during generation via a Token-Wise Hidden Streamer. These states are mapped into a compact latent space through a Sequence-to-Slot Latent Bridge and subsequently leveraged by an Evolving Latent Conditioner to drive a fast expert planner for real-time decision-making. This approach achieves, for the first time, token-level streaming of latent states from a slow reasoner to a rapid planner, significantly improving navigation success and social compliance on a newly introduced dynamic social VLN benchmark while maintaining computational efficiency and effectively reducing human collisions and unsafe behaviors caused by outdated observations.

Abstract

Vision-Language Navigation in dynamic, human-centric environments exposes a fundamental tension: linguistic reasoning is slow and deliberative, whereas safe, socially compliant planning should be instant and reactive. The resulting observation staleness is safety-critical: a maneuver chosen during inference can already be unsafe by the time it executes. We observe that, long before a VLM finishes its inference, its intermediate hidden states already encode action-relevant intent. We propose SPARK-VLN, a dual-system framework for dynamic social VLN that streams the slow VLM reasoner's knowledge to a fast flow-matching expert planner throughout token generation, providing fresh and evolving guidance during inference. This design is realized by three modules: a Token-Wise Hidden Streamer that extracts intermediate hidden states along the token generation process, a Sequence-to-Slot Latent Bridge that projects them into fixed-size latent slots, and an Evolving Latent Conditioner that infuses them into the expert planner. We also introduce a human-centric benchmark suite for dynamic social vision-language navigation that keeps pedestrians and the robot active throughout inference and reports navigation success, social compliance, human collisions, and explicit staleness statistics. Across these settings, SPARK-VLN mproves navigation success and social compliance while sustaining inference efficiency. Webpage: https://hutslib.github.io/SPARK-VLN/.

Problem

• Vision-Language Navigation
• Dynamic Environments
• Observation Staleness
• Social Compliance
• Real-time Planning

Innovation

• Latent Streaming
• Vision-Language Navigation
• Dynamic Social Navigation
• Flow-Matching Planner
• Token-Wise Reasoning