Beyond Symmetric Fusion: Exploiting Task-Dependent Modality Strengths for RGB-Event Small Object Detection

2026-08-02

AI Summary

Existing RGB-event fusion methods for small object detection typically adopt symmetric fusion strategies, overlooking the distinct strengths of each modality in localization versus classification tasks. To address this limitation, this work proposes AERODet, an asymmetric fusion framework that dynamically modulates the contributions of RGB and event modalities through a task-decoupling mechanism. Specifically, the framework integrates Scale-aware Uncertainty-based Reliability Estimation (SURE) and Task-Decoupled Semantic Refinement (TDSR), leveraging a Transformer architecture to enable asymmetric multimodal feature fusion. Evaluated on the FRED and NeRDD datasets, AERODet achieves state-of-the-art performance, outperforming the strongest baseline by 10.7 mAP points on the FRED challenge set.

Abstract

State-of-the-art RGB-Event detectors improve the detection of small, fast-moving objects by combining complementary features from RGB and Event data, yet they typically fuse the two modalities into a unified representation for both localization and classification. Such a task-symmetric design is inconsistent with the intuition that the two modalities should play different roles according to their task-specific strengths. To examine this issue, we conduct a modality-specific evaluation and find that the relative advantage of the two modalities reverses across tasks: Event data are substantially more effective for class-agnostic localization, whereas RGB data provide stronger category evidence within localized target regions. Motivated by this task-dependent asymmetry, we propose an Asymmetric Event-RGB Object Detection Transformer (AERODet). During class-agnostic localization, Scale-wise Uncertainty-aware Reliability Estimation (SURE) calculates the relative reliability of the two modalities from their objectness response heatmaps and accordingly calibrates their contributions when the decoder aggregates multimodal features. Once the candidate boxes are obtained, Task-Decoupled Semantic Refinement (TDSR) decouples classification from localization and uses RGB RoI features for fine-grained classification. Extensive experiments on FRED and NeRDD demonstrate that AERODet achieves state-of-the-art performance. In particular, it surpasses the strongest RGB-Event baseline by 10.7 mAP points on the FRED challenging split.

Problem

• RGB-Event fusion
• small object detection
• task-dependent modality strengths
• asymmetric fusion
• multimodal object detection

Innovation

• asymmetric fusion
• task-dependent modality strengths
• RGB-event object detection
• modality reliability estimation
• task-decoupled classification