Simulation-Based Performance Evaluation of 3D Object Detection Methods with Deep Learning for a LiDAR Point Cloud Dataset in a SOTIF-related Use Case

2025-03-05

International Conference on Vehicle Technology and Intelligent Transport Systems

AI Summary

Current SOTIF (Safety of the Intended Functionality) validation for autonomous driving lacks systematic assessment of LiDAR perception robustness under adverse weather conditions. Method: This work formally defines and models SOTIF-critical use cases, and constructs a high-fidelity simulated LiDAR point cloud dataset—comprising 547 frames across 21 weather-illumination combinations—using CARLA and PreScan. Within the MMDetection3D and OpenPCDet frameworks, it conducts a comprehensive performance evaluation of state-of-the-art 3D object detectors using AP and Recall metrics for cross-model comparison. Contribution/Results: Results reveal significant performance degradation under rain, fog, and low-light conditions, with AP dropping by up to 62%. To our knowledge, this is the first study to provide a reproducible, quantitative benchmark for evaluating LiDAR perception robustness in SOTIF contexts. The dataset and empirical findings bridge a critical gap in SOTIF validation, supporting algorithmic refinement and standardization efforts.

Technology Category

• Natural Language Processing: Safety and Robustness
• Computer Vision: Adversarial Attacks & Robustness
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization
• Security and Privacy: Large-scale security measurements

Abstract

Safety of the Intended Functionality (SOTIF) addresses sensor performance limitations and deep learning-based object detection insufficiencies to ensure the intended functionality of Automated Driving Systems (ADS). This paper presents a methodology examining the adaptability and performance evaluation of the 3D object detection methods on a LiDAR point cloud dataset generated by simulating a SOTIF-related Use Case. The major contributions of this paper include defining and modelling a SOTIF-related Use Case with 21 diverse weather conditions and generating a LiDAR point cloud dataset suitable for application of 3D object detection methods. The dataset consists of 547 frames, encompassing clear, cloudy, rainy weather conditions, corresponding to different times of the day, including noon, sunset, and night. Employing MMDetection3D and OpenPCDET toolkits, the performance of State-of-the-Art (SOTA) 3D object detection methods is evaluated and compared by testing the pre-trained Deep Learning (DL) models on the generated dataset using Average Precision (AP) and Recall metrics.

Problem

• Evaluates 3D object detection methods for LiDAR point clouds.
• Simulates diverse weather conditions for SOTIF-related use cases.
• Compares SOTA methods using AP and Recall metrics.

Innovation

• Simulated diverse weather conditions for LiDAR dataset
• Used MMDetection3D and OpenPCDET for 3D detection
• Evaluated models with AP and Recall metrics