Task-Aware Robotic Grasping by evaluating Quality Diversity Solutions through Foundation Models

2024-11-22

arXiv.org

AI Summary

Task-aware grasping requires joint semantic understanding and geometric reasoning, yet existing methods rely heavily on extensive task-specific annotations. This paper introduces the first semantic-geometric coupled representation framework for zero-shot, task-conditioned grasping: it jointly parses task intent and object structure via part-level semantic segmentation and large language models (LLMs), then employs quality-diversity (QD) optimization to generate interpretable and diverse grasping proposals. Key contributions include: (1) the first benchmark of task-specific ground-truth grasp regions; (2) an interpretable knowledge transfer mechanism from LLM-derived high-level task semantics to physically feasible grasping policies; and (3) state-of-the-art performance—73.6% weighted IoU across 65 task-object combinations—and strong user preference (88% favoring our method, *p* < 0.001).

Technology Category

• Computer Vision: Visual Reasoning & Symbolic Representations
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Task-aware robotic grasping is a challenging problem that requires the integration of semantic understanding and geometric reasoning. This paper proposes a novel framework that leverages Large Language Models (LLMs) and Quality Diversity (QD) algorithms to enable zero-shot task-conditioned grasp synthesis. The framework segments objects into meaningful subparts and labels each subpart semantically, creating structured representations that can be used to prompt an LLM. By coupling semantic and geometric representations of an object's structure, the LLM's knowledge about tasks and which parts to grasp can be applied in the physical world. The QD-generated grasp archive provides a diverse set of grasps, allowing us to select the most suitable grasp based on the task. We evaluated the proposed method on a subset of the YCB dataset with a Franka Emika robot. A consolidated ground truth for task-specific grasp regions is established through a survey. Our work achieves a weighted intersection over union (IoU) of 73.6% in predicting task-conditioned grasp regions in 65 task-object combinations. An end-to-end validation study on a smaller subset further confirms the effectiveness of our approach, with 88% of responses favoring the task-aware grasp over the control group. A binomial test shows that participants significantly prefer the task-aware grasp.

Problem

• Integrates semantic understanding and geometric reasoning for robotic grasping.
• Uses LLMs and QD algorithms for zero-shot task-conditioned grasp synthesis.
• Evaluates task-aware grasping on YCB dataset with Franka Emika robot.

Innovation

• Leverages LLMs and QD for zero-shot grasping
• Segments objects into semantically labeled subparts
• Uses QD-generated grasp archive for task suitability