S2R-Bench: A Sim-to-Real Evaluation Benchmark for Autonomous Driving

2025-05-24

AI Summary

Existing autonomous driving perception benchmarks rely heavily on simulation, failing to capture real-world degradation scenarios—such as extreme weather and sensor failures—leading to a severe sim-to-real performance gap. Method: We introduce S2R-Bench, the first real-scenario-oriented perception robustness benchmark: (i) it establishes a sim-to-real evaluation framework grounded in real-vehicle-collected camera/LiDAR corruption data; (ii) it systematically models four-dimensional correlated degradations—temporal, weather, illumination, and road conditions; and (iii) it proposes a consistency analysis framework bridging simulation and real-world results, alongside a standardized corruption protocol and open-source evaluation toolkit (GitHub). Results: Experiments reveal that mainstream perception models suffer an average 32.7% mAP drop under real-world corruptions, exposing critical deployment risks. S2R-Bench provides a reproducible, comparable, and operationally relevant safety evaluation standard for perception algorithms.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization
• Security and Privacy: Large-scale security measurements

Abstract

Safety is a long-standing and the final pursuit in the development of autonomous driving systems, with a significant portion of safety challenge arising from perception. How to effectively evaluate the safety as well as the reliability of perception algorithms is becoming an emerging issue. Despite its critical importance, existing perception methods exhibit a limitation in their robustness, primarily due to the use of benchmarks are entierly simulated, which fail to align predicted results with actual outcomes, particularly under extreme weather conditions and sensor anomalies that are prevalent in real-world scenarios. To fill this gap, in this study, we propose a Sim-to-Real Evaluation Benchmark for Autonomous Driving (S2R-Bench). We collect diverse sensor anomaly data under various road conditions to evaluate the robustness of autonomous driving perception methods in a comprehensive and realistic manner. This is the first corruption robustness benchmark based on real-world scenarios, encompassing various road conditions, weather conditions, lighting intensities, and time periods. By comparing real-world data with simulated data, we demonstrate the reliability and practical significance of the collected data for real-world applications. We hope that this dataset will advance future research and contribute to the development of more robust perception models for autonomous driving. This dataset is released on https://github.com/adept-thu/S2R-Bench.

Problem

• Evaluating safety and reliability of autonomous driving perception algorithms
• Addressing robustness gaps in simulated perception benchmarks
• Providing real-world scenario data for autonomous driving perception testing

Innovation

• Sim-to-Real benchmark for autonomous driving perception
• Diverse real-world sensor anomaly data collection
• Comprehensive evaluation under extreme conditions