RANa: Retrieval-Augmented Navigation

2025-04-04

AI Summary

This work addresses the challenge of experience reuse in embodied navigation, where agents typically discard historical knowledge after each episode. To overcome this limitation, we propose a retrieval-augmented navigation framework that enables dynamic retrieval and fusion of semantic and geometric information accumulated across prior tasks within the same environment—departing from conventional episodic memory reset paradigms. Methodologically, we design a multimodal database retrieval and context encoding mechanism grounded in vision foundation models, integrated with end-to-end reinforcement learning. We introduce the first navigation architecture supporting zero-shot transfer across both tasks and environments, and establish a novel benchmark explicitly evaluating historical information reuse. Experiments demonstrate significant performance gains on ObjectNav, ImageNav, and Instance-ImageNav tasks, validating strong generalization and zero-shot cross-task and cross-environment transfer capabilities.

Technology Category

• Computer Vision: Multi-modal Vision
• Search and Optimization: Learning to Search
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Methods for navigation based on large-scale learning typically treat each episode as a new problem, where the agent is spawned with a clean memory in an unknown environment. While these generalization capabilities to an unknown environment are extremely important, we claim that, in a realistic setting, an agent should have the capacity of exploiting information collected during earlier robot operations. We address this by introducing a new retrieval-augmented agent, trained with RL, capable of querying a database collected from previous episodes in the same environment and learning how to integrate this additional context information. We introduce a unique agent architecture for the general navigation task, evaluated on ObjectNav, ImageNav and Instance-ImageNav. Our retrieval and context encoding methods are data-driven and heavily employ vision foundation models (FM) for both semantic and geometric understanding. We propose new benchmarks for these settings and we show that retrieval allows zero-shot transfer across tasks and environments while significantly improving performance.

Problem

• Enhancing navigation by leveraging past episode data
• Integrating retrieval-augmented context into RL-trained agents
• Enabling zero-shot transfer across tasks via vision FMs

Innovation

• Retrieval-augmented agent trained with RL
• Database querying from previous episodes
• Vision foundation models for context encoding