Bootstrapping Object-level Planning with Large Language Models

2024-09-18

arXiv.org

AI Summary

Direct generation of PDDL goals or task sequences by large language models (LLMs) often yields semantically abstract, non-executable outputs, hindering integration with robot task and motion planning (TAMP). Method: We propose an LLM knowledge distillation framework that extracts object-level state-change knowledge via prompt engineering, constructs a function-oriented object network (FOON), and automatically compiles it into semantically aligned, executable PDDL subgoals. Contribution/Results: This FOON-PDDL joint representation establishes the first structured synergy between LLM-derived high-level semantics and classical planners’ action-object constraints. Evaluated on simulated pick-and-place tasks, our approach improves subgoal success rate by 37%, significantly enhancing planning feasibility and cross-task generalization.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

We introduce a new method that extracts knowledge from a large language model (LLM) to produce object-level plans, which describe high-level changes to object state, and uses them to bootstrap task and motion planning (TAMP). Existing work uses LLMs to directly output task plans or generate goals in representations like PDDL. However, these methods fall short because they rely on the LLM to do the actual planning or output a hard-to-satisfy goal. Our approach instead extracts knowledge from an LLM in the form of plan schemas as an object-level representation called functional object-oriented networks (FOON), from which we automatically generate PDDL subgoals. Our method markedly outperforms alternative planning strategies in completing several pick-and-place tasks in simulation.

Problem

• Extracts knowledge from LLMs for object-level planning.
• Generates PDDL subgoals from functional object-oriented networks.
• Improves task and motion planning in pick-and-place tasks.

Innovation

• Extracts knowledge from LLMs for object-level plans
• Uses FOON to generate PDDL subgoals automatically
• Outperforms in pick-and-place tasks via simulation