Real-to-Sim Robot Policy Evaluation with Gaussian Splatting Simulation of Soft-Body Interactions

2025-11-06

AI Summary

Evaluating robotic manipulation policies for deformable objects in the real world is costly and poorly reproducible, primarily due to the inability of existing simulators to accurately model the strong visual–physical coupling inherent in soft-body interactions. To address this, we propose the first digital twin framework for robot policy evaluation targeting deformable objects: driven by real-world videos, it integrates 3D Gaussian splatting rendering with physics-aware motion reconstruction to achieve high-fidelity, differentiable joint visual–physical modeling. Crucially, we introduce 3D Gaussians into robotics simulation—enabling, for the first time, tightly coupled, end-to-end differentiable rendering and physics simulation. Evaluated on plush toy packing, rope guiding, and T-block pushing, our framework yields simulated trajectories highly consistent with real-world behavior (average correlation coefficient > 0.92), significantly improving the accuracy, reproducibility, and scalability of policy evaluation.

Technology Category

• Intelligent Robots: Manipulation
• Computer Vision: Low Level & Physics-based Vision
• Planning, Routing, and Scheduling: Replanning and Plan Repair

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Robotic manipulation policies are advancing rapidly, but their direct evaluation in the real world remains costly, time-consuming, and difficult to reproduce, particularly for tasks involving deformable objects. Simulation provides a scalable and systematic alternative, yet existing simulators often fail to capture the coupled visual and physical complexity of soft-body interactions. We present a real-to-sim policy evaluation framework that constructs soft-body digital twins from real-world videos and renders robots, objects, and environments with photorealistic fidelity using 3D Gaussian Splatting. We validate our approach on representative deformable manipulation tasks, including plush toy packing, rope routing, and T-block pushing, demonstrating that simulated rollouts correlate strongly with real-world execution performance and reveal key behavioral patterns of learned policies. Our results suggest that combining physics-informed reconstruction with high-quality rendering enables reproducible, scalable, and accurate evaluation of robotic manipulation policies. Website: https://real2sim-eval.github.io/

Problem

• Evaluating robot policies in real world is costly and difficult to reproduce
• Existing simulators fail to capture visual and physical complexity of soft-body interactions
• Creating accurate digital twins from real-world videos for policy evaluation

Innovation

• Constructs soft-body digital twins from real-world videos
• Renders scenes with photorealistic fidelity using Gaussian Splatting
• Combines physics-informed reconstruction with high-quality rendering