Deep Multimodal Fusion Detection through Spatial Mask and Channel Fusion

2026-08-03

AI Summary

Existing multimodal fusion approaches often suffer from modality overfitting or excessive specialization, which compromises the generalization capability of cross-modal object detection. To address this, this work proposes an attention-driven complementary resampling framework that enhances cross-modal feature interaction through a shared channel-spatial attention mechanism. The method blurs modality boundaries via a semantic mask exchange strategy and introduces a learnable channel competition mechanism to enable dynamic, channel-wise sampling and aggregation. Notably, the framework operates without requiring explicit modality labels, thereby effectively promoting complementarity and generalization across modalities. Experimental results demonstrate that the proposed model consistently outperforms state-of-the-art methods on multiple benchmark datasets, confirming its efficacy and superiority.

Abstract

Deep multimodal fusion for object detection has demonstrated good performance through mining modal characteristics. However, existing feature-level fusion methods mainly weigh between two modalities and unify them in a unified representation space. This can lead to overfitting or over-specialization of the statistical properties of a single modality within a dual-backbone architecture. This paper proposes an Attention-Driven Complementarity Resampling framework for robust improvement of cross-modality object detection. Based on a shared channel spatial attention mechanism, we first introduce the semantic mask exchange to actively mix the boundaries of the modalities during the training phase, forcing the backbone network to learn generalized features without relying on fixed modal labels. Then we propose a learnable channel competition to sample and aggregate features in a channel-wise and learnable way. Our experiments on multiple datasets demonstrate that the proposed method is effective and yields competitive results among existing state-of-the-art approaches. The source code is provided in the supplementary material.

Problem

• multimodal fusion
• object detection
• feature-level fusion
• overfitting
• cross-modality

Innovation

• multimodal fusion
• spatial mask
• channel competition
• attention mechanism
• object detection