GSWorld: Closed-Loop Photo-Realistic Simulation Suite for Robotic Manipulation

2025-10-23

AI Summary

Addressing the longstanding challenge in robotic manipulation simulation—namely, the trade-off between photorealistic rendering and physically accurate interaction, as well as the poor reproducibility of sim2real policy transfer—this paper introduces GSDF, a closed-loop simulation framework integrating 3D Gaussian splatting rendering with a rigid-body physics engine. GSDF unifies Gaussian scene representations and URDF robot models via a novel asset format, enabling high-fidelity visual–physical co-simulation. It supports virtual teleoperation data collection, DAgger-style policy refinement, and zero-shot cross-domain transfer. Experiments demonstrate that GSDF enables end-to-end pixel-to-action sim2real policy training without physical robots, achieving high-precision benchmark evaluation across 40+ objects and multi-robot scenarios. To foster reproducibility and community advancement, the authors open-source both the GSDF simulation engine and its associated dataset.

Technology Category

• Intelligent Robots: Manipulation
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Planning, Routing, and Scheduling: Replanning and Plan Repair

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Agentic search

Abstract

This paper presents GSWorld, a robust, photo-realistic simulator for robotics manipulation that combines 3D Gaussian Splatting with physics engines. Our framework advocates"closing the loop"of developing manipulation policies with reproducible evaluation of policies learned from real-robot data and sim2real policy training without using real robots. To enable photo-realistic rendering of diverse scenes, we propose a new asset format, which we term GSDF (Gaussian Scene Description File), that infuses Gaussian-on-Mesh representation with robot URDF and other objects. With a streamlined reconstruction pipeline, we curate a database of GSDF that contains 3 robot embodiments for single-arm and bimanual manipulation, as well as more than 40 objects. Combining GSDF with physics engines, we demonstrate several immediate interesting applications: (1) learning zero-shot sim2real pixel-to-action manipulation policy with photo-realistic rendering, (2) automated high-quality DAgger data collection for adapting policies to deployment environments, (3) reproducible benchmarking of real-robot manipulation policies in simulation, (4) simulation data collection by virtual teleoperation, and (5) zero-shot sim2real visual reinforcement learning. Website: https://3dgsworld.github.io/.

Problem

• Developing photo-realistic robotic manipulation simulator with Gaussian Splatting
• Closing the loop for sim2real policy training without real robots
• Creating standardized asset format for diverse scene reconstruction

Innovation

• Combines 3D Gaussian Splatting with physics engines
• Introduces GSDF format for Gaussian-on-Mesh representation
• Enables sim2real policy training without real robots