SimEX: Simulation-Integrated Robotics AutoResearch

2026-09-30

AI Summary

This study addresses the challenge of enabling large language model (LLM) coding agents to efficiently and safely acquire physical robot control capabilities, noting that direct code generation lacks physical grounding while real-world trial-and-error remains prohibitively costly. To overcome this, we propose a simulation-integrated autonomous research framework that leverages the simulator as an iterative refinement laboratory rather than a mere data source. Specifically, we first develop general-purpose toolboxes within simulation, then combine code-as-policy with probe-based optimization algorithms to iteratively refine the simulator and diagnose failures through minimal real-world interactions. Applied to complex manipulation tasks such as towel folding, our approach enables robot skill acquisition without human demonstrations, requiring only ten minutes of real-world interaction. This significantly enhances both the efficiency and safety of transferring LLM agents to the physical world.

Abstract

Coding agents powered by large language models (LLMs) have shown remarkable abilities to autonomously reason about and achieve goals in the digital world. However, bringing this success to the physical world remains challenging. On the one hand, direct generation methods (e.g., Code as Policies) often suffer from the LLMs' insufficient understanding of robots and physical environments. On the other hand, iterative trial-and-error tuning in the physical world (e.g., physical autoresearch) induces significant experimental cost and safety concerns. We introduce SimEX: Simulation-Integrated Robotics AutoResearch, an autoresearch framework that tightly integrates simulated experimentation, enabling coding agents to efficiently acquire physical capabilities for controlling real robots. SimEX operates in two stages. First, the agent conducts open-ended probe-and-optimize iterations in simulation, developing a robot toolbox with robust and generalizable capabilities. Second, the agent adapts the toolbox and the simulator together through only a few physical trials: each trial corrects the simulator, and the corrected simulator is used to diagnose failures and screen candidate repairs. We evaluate SimEX extensively in sim-to-sim settings and on physical robots. On challenging real-world manipulation tasks including towel folding, barcode scanning, and plate manipulation, SimEX enables coding agents to efficiently acquire robot skills without any demonstration and with only 10 minutes of real-robot interaction. These results suggest that simulation can be a critical component in achieving physical intelligence, not only as a source of training data that must closely replicate the real world, but also as a roughly correct laboratory where a coding agent develops the knowledge and procedures needed to act on the robot. More details and robot videos at https://robo-simex.github.io/

Problem

• coding agents
• physical world robotics
• sim-to-real
• autoresearch
• large language models

Innovation

• Simulation-Integrated AutoResearch
• Coding Agents
• Sim-to-Real Adaptation
• Physical Intelligence
• Zero-Demonstration Learning