CFMW: Cross-modality Fusion Mamba for Multispectral Object Detection under Adverse Weather Conditions

2024-04-25

arXiv.org

AI Summary

Under adverse weather conditions—such as rain, snow, and haze—visible-infrared cross-modal object detection suffers severe performance degradation. To address this, we introduce SWVID, the first rigorously curated dual-modality dataset specifically designed for harsh-weather scenarios, and propose CFW, a lightweight and efficient model. Methodologically, CFW pioneers the integration of a weather-robust denoising diffusion model (WRDM) with the state-space model Mamba, enabling a weather-aware cross-modal fusion module (CFM). It further incorporates multispectral feature alignment and adaptive fusion to jointly suppress weather-induced degradations while ensuring low computational overhead and high modality robustness. Extensive experiments on SWVID and multiple public benchmarks demonstrate that CFW achieves state-of-the-art pedestrian detection accuracy and cross-weather generalization, balancing high precision with practical edge-deployment feasibility.

Technology Category

• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Cross-modality images that integrate visible-infrared spectra cues can provide richer complementary information for object detection. Despite this, existing visible-infrared object detection methods severely degrade in severe weather conditions. This failure stems from the pronounced sensitivity of visible images to environmental perturbations, such as rain, haze, and snow, which frequently cause false negatives and false positives in detection. To address this issue, we introduce a novel and challenging task, termed visible-infrared object detection under adverse weather conditions. To foster this task, we have constructed a new Severe Weather Visible-Infrared Dataset (SWVID) with diverse severe weather scenes. Furthermore, we introduce the Cross-modality Fusion Mamba with Weather-removal (CFMW) to augment detection accuracy in adverse weather conditions. Thanks to the proposed Weather Removal Diffusion Model (WRDM) and Cross-modality Fusion Mamba (CFM) modules, CFMW is able to mine more essential information of pedestrian features in cross-modality fusion, thus could transfer to other rarer scenarios with high efficiency and has adequate availability on those platforms with low computing power. To the best of our knowledge, this is the first study that targeted improvement and integrated both Diffusion and Mamba modules in cross-modality object detection, successfully expanding the practical application of this type of model with its higher accuracy and more advanced architecture. Extensive experiments on both well-recognized and self-created datasets conclusively demonstrate that our CFMW achieves state-of-the-art detection performance, surpassing existing benchmarks. The dataset and source code will be made publicly available at https://github.com/lhy-zjut/CFMW.

Problem

• Enhancing object detection robustness in adverse weather
• Reducing computational complexity in cross-modality fusion
• Addressing lack of diverse adverse weather datasets

Innovation

• Cross-modality Fusion Mamba for robust detection
• Perturbation-Adaptive Diffusion Model reconstructs features
• Efficient architecture design speeds up processing