A Survey of Robotic Navigation and Manipulation with Physics Simulators in the Era of Embodied AI

2025-05-01

AI Summary

To address performance degradation in sim-to-real transfer for embodied intelligence—caused by modeling discrepancies in physics simulators—this paper presents the first systematic, three-dimensional evaluation of mainstream engines (e.g., PyBullet, MuJoCo, Isaac Gym) along physical fidelity, task adaptability, and hardware constraints. We integrate cutting-edge techniques—including world models and geometrically equivariant networks—to establish a comprehensive benchmark featuring multi-task datasets, unified evaluation metrics, and an open-source platform. Furthermore, we propose a task-aware simulator selection framework that quantifies trade-offs among accuracy, real-time capability, differentiability, and deployment compatibility for navigation and manipulation tasks. Our contributions include an open-source evaluation repository and practical guidelines, providing both theoretical foundations and engineering evidence to reduce real-world training costs and enhance transfer robustness.

Technology Category

• Intelligent Robots: Embodied AI
• Search and Optimization: Sampling/Simulation-based Search
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization

Abstract

Navigation and manipulation are core capabilities in Embodied AI, yet training agents with these capabilities in the real world faces high costs and time complexity. Therefore, sim-to-real transfer has emerged as a key approach, yet the sim-to-real gap persists. This survey examines how physics simulators address this gap by analyzing their properties overlooked in previous surveys. We also analyze their features for navigation and manipulation tasks, along with hardware requirements. Additionally, we offer a resource with benchmark datasets, metrics, simulation platforms, and cutting-edge methods-such as world models and geometric equivariance-to help researchers select suitable tools while accounting for hardware constraints.

Problem

• Addressing sim-to-real gap in robotic navigation and manipulation
• Analyzing physics simulators' properties for Embodied AI tasks
• Providing benchmark resources for tool selection under hardware constraints

Innovation

• Physics simulators reduce sim-to-real gap
• Benchmark datasets aid tool selection
• World models enhance navigation and manipulation