Socratic Planner: Inquiry-Based Zero-Shot Planning for Embodied Instruction Following

2024-04-21

arXiv.org

AI Summary

To address the challenges of zero-shot planning difficulty and weak dynamic adaptability in Embodied Instruction Following (EIF), this paper proposes the first training-free, vision-driven zero-shot embodied planning framework. Methodologically, it decomposes natural language instructions into executable high-level sub-goal sequences via a self-questioning-and-answering mechanism; introduces a visually grounded real-time re-planning mechanism that dynamically refines plans based on visual feedback during interaction; and designs RelaxedHLP—a novel evaluation metric that quantifies high-level planning quality for the first time. Experiments on the ALFRED benchmark demonstrate state-of-the-art zero-shot and few-shot performance, particularly on complex tasks requiring multi-step reasoning and environment responsiveness. Visual feedback significantly improves re-planning accuracy, with consistent gains over existing methods.

Technology Category

• Humans and AI: Human-Aware Planning and Behavior Prediction
• Planning, Routing, and Scheduling: Planning with Language Models
• Intelligent Robots: Embodied AI

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Embodied Instruction Following (EIF) is the task of executing natural language instructions by navigating and interacting with objects in 3D environments. One of the primary challenges in EIF is compositional task planning, which is often addressed with supervised or in-context learning with labeled data. To this end, we introduce the Socratic Planner, the first zero-shot planning method that infers without the need for any training data. Socratic Planner first decomposes the instructions into substructural information of the task through self-questioning and answering, translating it into a high-level plan, i.e., a sequence of subgoals. Subgoals are executed sequentially, with our visually grounded re-planning mechanism adjusting plans dynamically through a dense visual feedback. We also introduce an evaluation metric of high-level plans, RelaxedHLP, for a more comprehensive evaluation. Experiments demonstrate the effectiveness of the Socratic Planner, achieving competitive performance on both zero-shot and few-shot task planning in the ALFRED benchmark, particularly excelling in tasks requiring higher-dimensional inference. Additionally, a precise adjustments in the plan were achieved by incorporating environmental visual information.

Problem

• Zero-shot planning for embodied instruction following tasks
• Handling unexpected obstacles during task execution
• Improving long-horizon task performance with complex inference

Innovation

• Self-QA-based zero-shot planning method
• Visually-grounded re-planning mechanism
• Large Language Model for subgoal generation