WNM-3D: A World Navigation Model with 3D Scene Conditioning for Closed-Loop VLN

2026-08-07

AI Summary

This work addresses the limitations of existing vision-and-language navigation methods, which lack explicit modeling of visual observation dynamics driven by actions and fail to effectively leverage geometrically aware historical scene information. The authors propose the first generative world model conditioned on 3D scene representations, where a frozen geometry encoder extracts 3D features from past observations, and a trainable Scene-to-Token Adapter compresses them into a fixed-length prefix. This prefix serves as a unified conditioning signal for the joint generation of future video frames and actions within a Diffusion Transformer framework. The model incorporates block-causal attention and is optimized through a combination of supervised fine-tuning, DAgger adaptation, and DanceGRPO closed-loop refinement. Evaluated on GN-Bench, it significantly outperforms strong baselines and its 2D-conditioned counterpart, demonstrating superior optical flow–action consistency and reduced visuomotor error in near-target scenarios.

Abstract

Recent vision-language navigation (VLN) systems increasingly adapt pretrained vision-language models (VLMs) into vision-language-action (VLA) policies that map egocentric observations and language instructions directly to navigation actions. Although semantically capable, such action-centric training does not explicitly model how the agent's visual observations should evolve under its predicted motion. Generative world-action models (WAMs) jointly predict future observations and actions, yet existing WAMs for continuous VLN do not condition joint future-view and action generation on geometry-aware representations inferred from the observed history. We present WNM-3D, a generative World Navigation Model with 3D scene conditioning for continuous VLN. To consolidate past observations into persistent scene context, a frozen feed-forward geometry encoder extracts geometry-aware representations from the monocular egocentric RGB history, and a trainable 3D Scene-to-Token Adapter converts them into a fixed-length prefix in the token space of the world-action Diffusion Transformer. Through block-causal attention, this prefix conditions every future video-action block, providing a shared geometric context for both future-view and action generation. We train WNM-3D through supervised world-action fine-tuning on A*-generated demonstrations, DAgger-style adaptation on policy-visited states, and DanceGRPO-based closed-loop policy optimization. Experiments on GN-Bench show that WNM-3D outperforms strong VLM-based navigation policies and its 2D-conditioned counterpart in closed-loop navigation. On a fixed near-goal evaluation set, WNM-3D also achieves higher flow-action consistency and lower visual-motion error.

Problem

• vision-language navigation
• world-action models
• 3D scene conditioning
• future observation prediction
• geometry-aware representation

Innovation

• 3D scene conditioning
• world-action model
• geometry-aware representation
• Diffusion Transformer
• closed-loop VLN