Growing with Your Embodied Agent: A Human-in-the-Loop Lifelong Code Generation Framework for Long-Horizon Manipulation Skills

2025-09-22

AI Summary

Existing LLM-driven robotic manipulation code generation methods face three key challenges: high noise in instruction translation, limited primitives and contextual awareness, and difficulty in reusing closed-loop feedback knowledge for long-horizon tasks—often leading to catastrophic forgetting. This paper proposes a human-in-the-loop lifelong code generation framework. Its core contributions are: (1) automatically encoding human corrections into structured, reusable skills; (2) integrating external memory with prompt-augmented retrieval-augmented generation (RAG) to enable dynamic retrieval and generalized invocation of correction knowledge; and (3) supporting continual learning and stable execution for ultra-long-horizon tasks (>20 steps). Evaluated across multiple simulation and real-world settings, our approach achieves a task success rate of 0.93 (up to +27% over baselines) and improves correction efficiency by 42% in terms of required refinement rounds, significantly outperforming state-of-the-art methods.

Technology Category

• Intelligent Robots: Manipulation
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Large language models (LLMs)-based code generation for robotic manipulation has recently shown promise by directly translating human instructions into executable code, but existing methods remain noisy, constrained by fixed primitives and limited context windows, and struggle with long-horizon tasks. While closed-loop feedback has been explored, corrected knowledge is often stored in improper formats, restricting generalization and causing catastrophic forgetting, which highlights the need for learning reusable skills. Moreover, approaches that rely solely on LLM guidance frequently fail in extremely long-horizon scenarios due to LLMs' limited reasoning capability in the robotic domain, where such issues are often straightforward for humans to identify. To address these challenges, we propose a human-in-the-loop framework that encodes corrections into reusable skills, supported by external memory and Retrieval-Augmented Generation with a hint mechanism for dynamic reuse. Experiments on Ravens, Franka Kitchen, and MetaWorld, as well as real-world settings, show that our framework achieves a 0.93 success rate (up to 27% higher than baselines) and a 42% efficiency improvement in correction rounds. It can robustly solve extremely long-horizon tasks such as "build a house", which requires planning over 20 primitives.

Problem

• LLM-based code generation struggles with noisy outputs and fixed primitives for robotic manipulation
• Existing methods suffer from catastrophic forgetting and cannot generalize corrected knowledge effectively
• Current approaches fail in extremely long-horizon tasks due to limited reasoning capabilities

Innovation

• Human-in-the-loop lifelong code generation framework
• Encodes corrections into reusable skills with external memory
• Uses Retrieval-Augmented Generation with hint mechanism