LLM-based Interactive Imitation Learning for Robotic Manipulation

2025-04-30

AI Summary

Imitation learning (IL) and interactive imitation learning (IIL) in robotic manipulation heavily rely on human teachers, resulting in high annotation costs and scalability limitations. Method: This paper proposes LLM-iTeach, the first framework to deeply integrate large language models (LLMs) into the IIL closed loop as scalable, low-cost, human-like interactive teachers. It employs hierarchical prompt engineering to guide LLMs in generating executable Python policy code and introduces a similarity-driven dynamic feedback mechanism to enable efficient teacher–learner interaction, coupled with behavior cloning (BC) for policy training. Results: Evaluated across diverse robotic manipulation tasks, LLM-iTeach matches or surpasses the human-teacher-driven CEILing baseline while significantly outperforming standard BC. These results demonstrate strong generalization and practical efficacy, effectively breaking the long-standing dependency of IL/IIL on real-time human intervention.

Technology Category

• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Intelligent Robots: Manipulation
• Humans and AI: Human-in-the-loop Machine Learning

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Recent advancements in machine learning provide methods to train autonomous agents capable of handling the increasing complexity of sequential decision-making in robotics. Imitation Learning (IL) is a prominent approach, where agents learn to control robots based on human demonstrations. However, IL commonly suffers from violating the independent and identically distributed (i.i.d) assumption in robotic tasks. Interactive Imitation Learning (IIL) achieves improved performance by allowing agents to learn from interactive feedback from human teachers. Despite these improvements, both approaches come with significant costs due to the necessity of human involvement. Leveraging the emergent capabilities of Large Language Models (LLMs) in reasoning and generating human-like responses, we introduce LLM-iTeach -- a novel IIL framework that utilizes an LLM as an interactive teacher to enhance agent performance while alleviating the dependence on human resources. Firstly, LLM-iTeach uses a hierarchical prompting strategy that guides the LLM in generating a policy in Python code. Then, with a designed similarity-based feedback mechanism, LLM-iTeach provides corrective and evaluative feedback interactively during the agent's training. We evaluate LLM-iTeach against baseline methods such as Behavior Cloning (BC), an IL method, and CEILing, a state-of-the-art IIL method using a human teacher, on various robotic manipulation tasks. Our results demonstrate that LLM-iTeach surpasses BC in the success rate and achieves or even outscores that of CEILing, highlighting the potential of LLMs as cost-effective, human-like teachers in interactive learning environments. We further demonstrate the method's potential for generalization by evaluating it on additional tasks. The code and prompts are provided at: https://github.com/Tubicor/LLM-iTeach.

Problem

• LLM-iTeach reduces human dependency in robotic imitation learning
• Hierarchical prompting enables LLM to generate Python-based policies
• Similarity feedback improves agent performance in manipulation tasks

Innovation

• LLM-based interactive teacher for robotic learning
• Hierarchical prompting for policy generation in Python
• Similarity-based feedback mechanism for corrective training