REHEARSE-3D: A Multi-modal Emulated Rain Dataset for 3D Point Cloud De-raining

2025-04-30

AI Summary

Heavy rainfall introduces spurious points into LiDAR point clouds, severely degrading the reliability of autonomous driving perception. To address this, we introduce RainSim3D—the first large-scale, point-level annotated, multimodal (256-channel LiDAR + 4D radar) rainy-weather simulation dataset, covering both daytime and nighttime scenarios, with physically grounded raindrop modeling and spatiotemporally synchronized calibration. We propose a novel point-wise rain feature injection mechanism and establish the first cross-modal LiDAR-radar fusion framework for 3D point cloud deraining. Furthermore, we construct the first unified benchmark to systematically evaluate both statistical and deep learning-based deraining methods. All data, annotations, and models will be publicly released, providing foundational resources for robust perception research under adverse weather conditions.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Web Mining and Content Analysis: Web data generation and simulation

Abstract

Sensor degradation poses a significant challenge in autonomous driving. During heavy rainfall, the interference from raindrops can adversely affect the quality of LiDAR point clouds, resulting in, for instance, inaccurate point measurements. This, in turn, can potentially lead to safety concerns if autonomous driving systems are not weather-aware, i.e., if they are unable to discern such changes. In this study, we release a new, large-scale, multi-modal emulated rain dataset, REHEARSE-3D, to promote research advancements in 3D point cloud de-raining. Distinct from the most relevant competitors, our dataset is unique in several respects. First, it is the largest point-wise annotated dataset, and second, it is the only one with high-resolution LiDAR data (LiDAR-256) enriched with 4D Radar point clouds logged in both daytime and nighttime conditions in a controlled weather environment. Furthermore, REHEARSE-3D involves rain-characteristic information, which is of significant value not only for sensor noise modeling but also for analyzing the impact of weather at a point level. Leveraging REHEARSE-3D, we benchmark raindrop detection and removal in fused LiDAR and 4D Radar point clouds. Our comprehensive study further evaluates the performance of various statistical and deep-learning models. Upon publication, the dataset and benchmark models will be made publicly available at: https://sporsho.github.io/REHEARSE3D.

Problem

• Addressing LiDAR point cloud degradation caused by heavy rainfall
• Providing a large-scale multi-modal dataset for 3D de-raining research
• Benchmarking raindrop detection/removal in fused LiDAR-4D Radar data

Innovation

• Largest point-wise annotated rain dataset
• High-resolution LiDAR with 4D Radar fusion
• Rain-characteristic info for noise modeling