LLM-Craft: Robotic Crafting of Elasto-Plastic Objects with Large Language Models

2024-06-12

arXiv.org

AI Summary

This work addresses robotic plastic shaping of elastoplastic deformable objects (e.g., clay), proposing the first large language model (LLM)-driven high-level shape reasoning framework. Unlike conventional low-level motion planning, our method jointly encodes object states and actions as geometric-semantic symbol sequences, enabling the LLM to directly infer mappings from target shapes to deformation actions. By integrating physics-informed deformation modeling with iterative closed-loop reasoning, the framework synergistically optimizes abstract shape representation and execution feedback. Experiments on a soft robotic platform demonstrate precise fabrication of geometric shapes (e.g., letters) and successful generalization to fuzzy semantic commands such as “thinner” or “convex/concave.” Results validate the LLM’s capacity for causal modeling of elastoplastic deformation processes and effective high-level planning—marking a significant step toward semantically grounded, adaptive robotic shaping.

Technology Category

• Intelligent Robots: Manipulation
• Planning, Routing, and Scheduling: Planning with Language Models
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

When humans create sculptures, we are able to reason about how geometrically we need to alter the clay state to reach our target goal. We are not computing point-wise similarity metrics, or reasoning about low-level positioning of our tools, but instead determining the higher-level changes that need to be made. In this work, we propose LLM-Craft, a novel pipeline that leverages large language models (LLMs) to iteratively reason about and generate deformation-based crafting action sequences. We simplify and couple the state and action representations to further encourage shape-based reasoning. To the best of our knowledge, LLM-Craft is the first system successfully leveraging LLMs for complex deformable object interactions. Through our experiments, we demonstrate that with the LLM-Craft framework, LLMs are able to successfully reason about the deformation behavior of elasto-plastic objects. Furthermore, we find that LLM-Craft is able to successfully create a set of simple letter shapes. Finally, we explore extending the framework to reaching more ambiguous semantic goals, such as"thinner"or"bumpy". For videos please see our website: https://sites.google.com/andrew.cmu.edu/llmcraft.

Problem

• Leveraging LLMs for robotic deformation-based crafting actions
• Reasoning about high-level geometric changes in elasto-plastic objects
• Simplifying state-action representations for shape-based crafting

Innovation

• Leverages LLMs for robotic crafting
• Simplifies state and action representations
• First system for deformable object interactions