Diffusion as Reasoning: Enhancing Object Goal Navigation with LLM-Biased Diffusion Model

2024-10-29

arXiv.org

AI Summary

To address insufficient semantic prior exploitation and poor cross-scene generalization in Object Navigation (ObjectNav) within unknown environments, this paper proposes a “Diffusion-as-Inference” framework: conditioned on the explored semantic map, a conditional diffusion model generates semantic predictions for unobserved regions to localize target objects. We introduce two key innovations: (1) a global-goal-guided noise scheduling mechanism that aligns denoising trajectories with navigation objectives, and (2) an LLM-distilled local commonsense bias that injects spatial-semantic priors into the diffusion process. This work is the first to adapt the diffusion model paradigm to spatial-semantic reasoning tasks and the first to propose an LLM-driven commonsense bias mechanism. Evaluated on Gibson and MatterPort3D (MP3D), our method achieves significant improvements in ObjectNav success rate and SPL, while boosting semantic map generation accuracy by 23.6% and demonstrating robust generalization to unseen object categories.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Search and Optimization: Learning to Search
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

The Object Goal Navigation (ObjectNav) task requires the agent to navigate to a specified target in an unseen environment. Since the environment layout is unknown, the agent needs to perform semantic reasoning to infer the potential location of the target, based on its accumulated memory of the environment during the navigation process. Diffusion models have been shown to be able to learn the distribution relationships between features in RGB images, and thus generate new realistic images.In this work, we propose a new approach to solving the ObjectNav task, by training a diffusion model to learn the statistical distribution patterns of objects in semantic maps, and using the map of the explored regions during navigation as the condition to generate the map of the unknown regions, thereby realizing the semantic reasoning of the target object, i.e., diffusion as reasoning (DAR). Meanwhile, we propose the global target bias and local LLM bias methods, where the former can constrain the diffusion model to generate the target object more effectively, and the latter utilizes the common sense knowledge extracted from the LLM to improve the generalization of the reasoning process. Based on the generated map in the unknown region, the agent sets the predicted location of the target as the goal and moves towards it. Experiments on Gibson and MP3D show the effectiveness of our method.

Problem

• Enhancing ObjectNav by integrating contextual relation reasoning
• Improving map quality in unexplored areas using known information
• Leveraging LLM-based knowledge for room-aware object distribution guidance

Innovation

• Diffusion model learns object distribution patterns
• LLM-based Room Guidance enhances object distributions
• Generates unknown region maps from explored areas