AutoMisty: A Multi-Agent LLM Framework for Automated Code Generation in the Misty Social Robot

2025-03-09

AI Summary

Existing social robot programming interfaces remain inaccessible to non-programmers, hindering widespread adoption of customizable, open-domain human-robot interaction. Method: We propose the first multi-LLM agent framework tailored for non-programming users to enable end-to-end generation of executable code for Misty robots from natural language instructions. Our approach employs a four-agent collaborative architecture for task decomposition, assignment, solution synthesis, and result integration, augmented by a dual-layer optimization mechanism—self-reflective reasoning and human-in-the-loop correction—to ensure transparency, controllability, and iterative refinement. Contribution/Results: Technically, the framework integrates multi-agent systems, large language models, task composition, human feedback alignment, and deep Misty API integration. Evaluated on a four-tier real-robot benchmark, our method significantly outperforms ChatGPT-4o and o1 in both code generation quality and execution accuracy, achieving— for the first time—fully automated, high-fidelity, user-controllable open-domain social interaction.

Technology Category

• Intelligent Robots: Human-Robot Interaction
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Agentic search

Abstract

The social robot's open API allows users to customize open-domain interactions. However, it remains inaccessible to those without programming experience. In this work, we introduce AutoMisty, the first multi-agent collaboration framework powered by large language models (LLMs), to enable the seamless generation of executable Misty robot code from natural language instructions. AutoMisty incorporates four specialized agent modules to manage task decomposition, assignment, problem-solving, and result synthesis. Each agent incorporates a two-layer optimization mechanism, with self-reflection for iterative refinement and human-in-the-loop for better alignment with user preferences. AutoMisty ensures a transparent reasoning process, allowing users to iteratively refine tasks through natural language feedback for precise execution. To evaluate AutoMisty's effectiveness, we designed a benchmark task set spanning four levels of complexity and conducted experiments in a real Misty robot environment. Extensive evaluations demonstrate that AutoMisty not only consistently generates high-quality code but also enables precise code control, significantly outperforming direct reasoning with ChatGPT-4o and ChatGPT-o1. All code, optimized APIs, and experimental videos will be publicly released through the webpage: https://wangxiaoshawn.github.io/AutoMisty.html

Problem

• Enables code generation for Misty robot via natural language.
• Uses multi-agent LLM framework for task decomposition and synthesis.
• Improves code quality and control over direct LLM reasoning.

Innovation

• Multi-agent LLM framework for code generation
• Two-layer optimization with self-reflection
• Natural language feedback for task refinement