UnitedVLN: Generalizable Gaussian Splatting for Continuous Vision-Language Navigation

2024-11-25

arXiv.org

AI Summary

To address insufficient localization and planning robustness in Vision-and-Language Navigation in Continuous Environments (VLN-CE)—caused by visual occlusion and unstructured paths—this paper introduces the first 3D Gaussian Splatting (3DGS)-based pretraining paradigm tailored for VLN. Our method jointly optimizes geometry, appearance, and semantics to render high-fidelity 360° images and dense semantic features in a unified framework. We propose a novel search-then-query sampling strategy and a decoupled-then-unified rendering mechanism, enabling the first fine-grained co-modeling of appearance and high-level semantics for continuous navigation. Integrating NeRF-enhanced sampling with cross-modal alignment, our approach achieves new state-of-the-art performance across mainstream VLN-CE benchmarks, significantly improving both navigation success rate and path fidelity. Crucially, it demonstrates strong generalization to unseen scenes and instructions.

Technology Category

• Computer Vision: 3D Computer Vision
• Search and Optimization: Sampling/Simulation-based Search
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Vision-and-Language Navigation (VLN), where an agent follows instructions to reach a target destination, has recently seen significant advancements. In contrast to navigation in discrete environments with predefined trajectories, VLN in Continuous Environments (VLN-CE) presents greater challenges, as the agent is free to navigate any unobstructed location and is more vulnerable to visual occlusions or blind spots. Recent approaches have attempted to address this by imagining future environments, either through predicted future visual images or semantic features, rather than relying solely on current observations. However, these RGB-based and feature-based methods lack intuitive appearance-level information or high-level semantic complexity crucial for effective navigation. To overcome these limitations, we introduce a novel, generalizable 3DGS-based pre-training paradigm, called UnitedVLN, which enables agents to better explore future environments by unitedly rendering high-fidelity 360 visual images and semantic features. UnitedVLN employs two key schemes: search-then-query sampling and separate-then-united rendering, which facilitate efficient exploitation of neural primitives, helping to integrate both appearance and semantic information for more robust navigation. Extensive experiments demonstrate that UnitedVLN outperforms state-of-the-art methods on existing VLN-CE benchmarks.

Problem

• Enhance navigation in continuous environments using 3DGS-based pre-training.
• Integrate appearance and semantic information for robust navigation.
• Outperform state-of-the-art methods on VLN-CE benchmarks.

Innovation

• Generalizable 3DGS-based pre-training paradigm
• Search-then-query sampling for neural primitives
• Separate-then-united rendering for robust navigation