ZeroBot: Learning From Scratch in Minutes With Generative Real2Sim

2026-04-01

IEEE Robotics and Automation Letters

AI Summary

This study addresses the challenge of rapid robot skill acquisition under zero-demonstration, zero-pretraining, and unknown-model conditions by proposing a parallel reinforcement learning framework based on generative simulation. The method leverages Image-to-3D models and Real2Sim techniques to automatically generate 3D meshes from images and construct corresponding simulation environments. By integrating geometric priors with value function-guided contact state sampling, it enables minute-scale policy training through large-scale parallel reinforcement learning. Experimental results demonstrate that the proposed system achieves an average training time of only 119 seconds while attaining an 87% success rate on real-world physical tasks. This work presents a novel paradigm for efficient robotic learning in open-ended scenarios.

Abstract

We present ZeroBot, a real2sim framework for learning a robot manipulation task from scratch in minutes under challenging conditions: zero human demonstrations, zero policy pre-training, and zero known object models. Given only a single view of an object and a goal pose for that object, ZeroBot uses image-to-3D generative models to obtain a complete object mesh, which is used in simulation for large-scale parallel reinforcement learning. To accelerate training, we introduce an action space which leverages the generated geometry and learned value function to sample states involving robot-object contact. When evaluated on real-world tasks including grasping, pushing, articulated object interaction, and multi-stage manipulation, ZeroBot achieves an 87% success rate with an average training time of 119 seconds. These results show the value of using image-to-3D models in a real2sim framework for rapid, autonomous robot learning.

Problem

• robot manipulation
• real2sim
• zero-shot learning
• reinforcement learning
• image-to-3D

Innovation

• Real2Sim
• Generative Image-to-3D
• Reinforcement Learning
• Zero-shot Manipulation
• Action Space Design