TeraSim-World: Worldwide Safety-Critical Data Synthesis for End-to-End Autonomous Driving

2025-09-16

AI Summary

To address the scarcity of real safety-critical data, high cost and risk of on-road testing, and significant sim-to-real domain gap in end-to-end autonomous driving, this paper proposes the first fully pipeline framework for synthesizing safety-critical scenarios adaptable to arbitrary geographic locations. Methodologically, it integrates high-definition maps, naturalistic driving trajectories, and traffic demand modeling to construct a geography-aware adversarial scenario generation pipeline; it further innovatively incorporates the Cosmos-Drive video generation model to achieve street-level semantic alignment and photorealistic sensor rendering. Experiments generate high-fidelity synthetic data spanning diverse climates, terrains, and traffic densities. Results demonstrate substantial reduction in the sim-to-real gap across multiple benchmarks—e.g., detection error reduced by 32.7%—effectively supporting end-to-end model training and robustness validation.

Technology Category

• Computer Vision: Generative Adversarial Networks (GANs) for Vision
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Web Mining and Content Analysis: Web data generation and simulation
• Security and Privacy: Large-scale security measurements

Abstract

Safe and scalable deployment of end-to-end (E2E) autonomous driving requires extensive and diverse data, particularly safety-critical events. Existing data are mostly generated from simulators with a significant sim-to-real gap or collected from on-road testing that is costly and unsafe. This paper presents TeraSim-World, an automated pipeline that synthesizes realistic and geographically diverse safety-critical data for E2E autonomous driving at anywhere in the world. Starting from an arbitrary location, TeraSim-World retrieves real-world maps and traffic demand from geospatial data sources. Then, it simulates agent behaviors from naturalistic driving datasets, and orchestrates diverse adversities to create corner cases. Informed by street views of the same location, it achieves photorealistic, geographically grounded sensor rendering via the frontier video generation model Cosmos-Drive. By bridging agent and sensor simulations, TeraSim-World provides a scalable and critical~data synthesis framework for training and evaluation of E2E autonomous driving systems.

Problem

• Synthesizes realistic safety-critical data for autonomous driving
• Bridges sim-to-real gap with geographically diverse scenarios
• Generates photorealistic corner cases for E2E system evaluation

Innovation

• Automated pipeline synthesizes realistic safety-critical data
• Simulates agent behaviors from naturalistic driving datasets
• Achieves photorealistic rendering via Cosmos-Drive model