Real2Gym: Building Gyms from Videos, Bringing Skills to Robots

2026-09-29

AI Summary

This study addresses the challenge of translating real-world videos into reusable robotic skills, which is hindered by difficulties in visual alignment, physical interaction, and experiential learning. To overcome these limitations, this work proposes a Real2Sim2Real framework that introduces a pioneering agent-based video-to-simulation construction mechanism for reconstructing editable scenes from videos. Manipulation policies are distilled through code generation and physics engine validation, while zero-shot sim-to-real transfer is achieved via a shared perception-control interface without weight updates. Empirical evaluations demonstrate that the proposed approach surpasses GPT-6 Astra Direct Mode by 16.7% in simulation success rate, reduces execution tokens by approximately 74.9%, and improves real-world success rates by 33.3%.

Abstract

Real-world videos provide rich demonstrations of manipulation, but turning them into reusable robot skills requires visually aligned environments, executable physical interactions, and mechanisms for learning from experience. We introduce Real2Gym, an agentic Real2Sim2Real framework that turns human and robot demonstrations into interactive simulation gyms and brings skills acquired in simulation to physical robots. The Real2Sim module reconstructs editable scenes, aligns objects and cameras with the input, validates demonstrated or retargeted actions through native physics execution, and generates task-conditioned variations with action-feasibility checks. Within these environments, the agent generates executable code for manipulation stages, observes their outcomes, and distills successful attempts and failures into reusable task procedures, object-relative motions, and recovery strategies. Through a shared perception-and-control interface, these skills guide subsequent execution in simulation and on real robots, with motions adapted to current observations and no updates to the underlying model weights. Extensive evaluations demonstrate that Real2Gym enables high-fidelity simulation environment reconstruction, outperforming GPT-6 Astra Direct Mode by 16.7% in success rate with approximately 74.9% fewer policy-execution tokens across these environments, while exceeding it by 33.3% in physical robot execution success rate across four tasks on a real Franka robot.

Problem

• Real-to-Sim-to-Real
• robot manipulation
• skill transfer
• simulation environment reconstruction
• learning from demonstrations

Innovation

• Real2Sim2Real
• Agentic framework
• Simulation gym generation
• Skill distillation
• Zero-shot sim-to-real transfer