World Translation: Minimizing Sim-to-Real Gap with Backward Dynamics Extraction and Unpaired Domain Translation

2026-07-20

AI Summary

This work addresses the challenge of sim-to-real policy transfer failures caused by unobservable dynamics—such as abrupt contacts—by introducing an inverse dynamics extraction mechanism that recovers implicit dynamical information from real-world transition data. The approach formulates dynamics transfer between simulation and reality as an unpaired domain translation task, preserving domain-specific styles while enabling effective cross-domain adaptation. By integrating physics-based simulation with real robot data, it overcomes the limitations of conventional methods that rely solely on observed history to infer latent variables. Experimental validation across humanoid, quadrupedal, and robotic arm platforms demonstrates substantially improved dynamics modeling accuracy, particularly in scenarios where observation history is insufficient or misleading. Real-world trials on the Go2 quadruped further confirm a marked enhancement in policy transfer performance.

Abstract

The gap between simulation and reality remains a fundamental challenge in deploying simulation-trained robotic policies in the real world. Real-to-sim methods narrow this gap from the real side, learning transition dynamics from real data to build a more realistic digital world. Learned dynamics models are their dominant instance. Such methods, however, face a partial observability problem: the same observation may branch to different transitions due to unobservable factors. Existing methods assume these factors can be recovered from observation history. However, this may fail whenever observation history is uninformative, such as a sudden contact event with no prior warning. To address this limitation, we propose \textit{World Translation}, which exploits a complementary strength of simulators and learned dynamics. Simulators are deterministic but physically imperfect, while learned models are accurate but underdetermined under partial observability. Rather than predicting transitions forward from history, we extract the unobservable dynamics information backward from an observed transition, then translate this feature across simulation and reality as an unpaired domain-translation problem that preserves dynamics content while transferring domain style. Experiments across humanoid, quadruped, and manipulator platforms show that our method achieves more accurate dynamics modeling than baselines, with the largest gains when unobservable factors cannot be recovered from observation history. Real-robot deployment on Go2 quadruped confirms improved policy transfer.

Problem

• sim-to-real gap
• partial observability
• dynamics modeling
• domain translation
• robotic policy transfer

Innovation

• World Translation
• backward dynamics extraction
• unpaired domain translation
• sim-to-real transfer
• partial observability