RegisterBridgeMM: A Register-Centric Framework for RGB-Infrared Object Detection

2026-08-05

AI Summary

This work addresses the challenge of cross-modal fusion in RGB-infrared object detection under varying illumination, diverse weather conditions, and complex scenes. The authors propose a novel three-stage fusion framework leveraging pre-trained registers: Aggregate preserves modality-specific register summaries, Bridge enables efficient cross-modal interaction through bidirectional register-to-patch reading, and Project transforms register information into spatially adaptive feature calibration. Notably, this approach introduces pre-trained registers as compact communication mediators, avoiding dense patch-wise interactions while retaining pre-trained representational capacity, and incorporates a consensus residual modulation mechanism. Evaluated on four benchmarks—LLVIP, M3FD, DroneVehicle, and FLIR-Aligned—the method achieves state-of-the-art mAP50-95 performance under backbone-freezing settings.

Abstract

RGB-infrared (RGB-IR) object detection benefits from complementary visible and thermal cues, but effective fusion remains challenging under illumination changes, weather variation, and cluttered scenes. Existing RGB-IR fusion methods often trade expressive patch-level interaction for lighter but more constrained adaptation mechanisms. We empirically observe that pretrained register tokens contain both modality-shared and modality-specific information on paired RGB-IR inputs, suggesting that they can serve as a compact substrate for cross-modal communication. Building on this observation, we propose RegisterBridgeMM, a register-mediated fusion framework organized as a three-stage register lifecycle. Aggregate preserves per-modality register summarization inherited from pretraining; Bridge performs bidirectional register-to-patch reading with consensus-residual regulation; and Project translates the resulting register summary into spatially adaptive calibration of patch features. This register pathway avoids dense patch-to-patch cross-modal interaction while preserving the pretrained patch representation. With both backbone streams frozen, RegisterBridgeMM achieves the highest mAP50-95 among the evaluated methods on all four benchmarks: LLVIP, M3FD, DroneVehicle, and FLIR-Aligned.

Problem

• RGB-infrared object detection
• cross-modal fusion
• illumination changes
• weather variation
• cluttered scenes

Innovation

• register-based fusion
• RGB-infrared object detection
• cross-modal communication
• pretrained register tokens
• spatially adaptive calibration