Getting Out and Getting Back: World and Behavior Grounding in Real2Sim2Real Co-Training

2026-09-30

AI Summary

This study addresses the unclear impact of world fidelity and human behavioral similarity in simulated data on policy performance. To this end, we propose a Real2Sim2Real co-training framework that decouples and independently modulates world grounding and behavior grounding during simulation data generation. By incorporating dynamic dexterous manipulation tasks and latent space analysis, this work reveals the complementary mechanisms through which both forms of grounding enhance policy performance. Experimental results demonstrate that full grounding improves the success rate from 52% to 86%, validating the critical value of highly grounded simulated data for foundation model co-training.

Abstract

Simulation can expand scarce real demonstrations for co-training, yet how world fidelity and similarity to human behavior affect policy performance remains unclear. We distinguish world grounding, which aligns simulation with the real system, and behavior grounding, which aligns simulated trajectories with human motion. We build a real2sim2real pipeline that varies these axes independently to generate data for co-training. On a dynamic dexterous pick-and-sort task, fully grounded co-training raises success from 52% to 86%; averaged across configurations, world grounding improves success by 18 percentage points and behavior grounding by 10. Deployed policies behave like a mixture of real-derived and simulation-derived policies, imitating real demonstrations in covered states and relying on simulated behavior elsewhere, which we examine through latent-space analysis. Together, these results suggest complementary roles: world grounding lets policies use simulated experience beyond real-data coverage, while behavior grounding matters mainly when world grounding is imperfect. Grounded simulation remains beneficial when co-training foundation models.

Problem

• Real2Sim2Real
• co-training
• world grounding
• behavior grounding
• policy performance

Innovation

• Real2Sim2Real
• World Grounding
• Behavior Grounding
• Co-training
• Dexterous Manipulation