GaussGym: An open-source real-to-sim framework for learning locomotion from pixels

2025-10-17

AI Summary

This work addresses the longstanding challenge of simultaneously achieving high visual fidelity and computational efficiency in physics-based simulation. We propose the first end-to-end framework that integrates 3D Gaussian Splatting (3DGS) as a plug-and-play neural renderer into vectorized physics simulators (e.g., IsaacGym). Our method enables high-fidelity, pixel-accurate simulation at over 100,000 steps per second, facilitating rapid large-scale scene construction and high-throughput policy training on consumer-grade GPUs. By unifying differentiable neural rendering with rigid-body physics simulation, we significantly enhance the semantic richness and geometric accuracy of visual observations for navigation and control decisions, while enabling robust sim-to-real transfer. Extensive experiments demonstrate strong generalization across diverse legged locomotion tasks and validate effective real-world deployment.

Technology Category

• Computer Vision: Low Level & Physics-based Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Systems and Infrastructure for Web, Mobile and WoT: Web applications in cross-disciplinary domains and verticals such as mixed reality, smart cities, and digital health

Abstract

We present a novel approach for photorealistic robot simulation that integrates 3D Gaussian Splatting as a drop-in renderer within vectorized physics simulators such as IsaacGym. This enables unprecedented speed -- exceeding 100,000 steps per second on consumer GPUs -- while maintaining high visual fidelity, which we showcase across diverse tasks. We additionally demonstrate its applicability in a sim-to-real robotics setting. Beyond depth-based sensing, our results highlight how rich visual semantics improve navigation and decision-making, such as avoiding undesirable regions. We further showcase the ease of incorporating thousands of environments from iPhone scans, large-scale scene datasets (e.g., GrandTour, ARKit), and outputs from generative video models like Veo, enabling rapid creation of realistic training worlds. This work bridges high-throughput simulation and high-fidelity perception, advancing scalable and generalizable robot learning. All code and data will be open-sourced for the community to build upon. Videos, code, and data available at https://escontrela.me/gauss_gym/.

Problem

• Developing photorealistic robot simulation with 3D Gaussian Splatting
• Enabling high-speed locomotion learning from visual inputs
• Bridging simulation fidelity with scalable training environments

Innovation

• Integrates 3D Gaussian Splatting as renderer for simulation
• Enables high-speed simulation with visual fidelity
• Uses iPhone scans and datasets for training worlds