DifferSeg: Towards Diverse Multimodal Binary Segmentation via Differential Perception and Frequency Guidance

2026-06-07

AI Summary

Existing multimodal binary segmentation methods struggle to adaptively handle modality discrepancies and complementarity, and their decoding strategies often fail to effectively balance high- and low-frequency information. To address these limitations, this work proposes DifferSeg, a novel framework featuring a Difference-aware Fusion (DPF) module and a Frequency-Guided Decoder (FGD). The DPF module employs learnable differential operators to achieve adaptive alignment and residual fusion of modality-specific features, thereby mitigating modality mismatch and redundancy. Meanwhile, the FGD incorporates cross-frequency interactions and a multi-path upsampling mechanism to jointly optimize high- and low-frequency representations, preserving both boundary details and semantic consistency. Evaluated across 29 public datasets and 18 downstream tasks, DifferSeg outperforms 67 state-of-the-art methods, demonstrating significantly improved segmentation accuracy and cross-domain generalization capability.

Abstract

In many binary segmentation tasks, most multimodal methods rely on fixed feature concatenation for cross-modal interaction and straightforward decoder designs dominated by low-frequency semantics. %ToDO: % However, they ignore two key challenges: one is the lack of an adaptive mechanism to handle modality discrepancies and complementarity, and the other is the absence of an efficient decoding strategy to balance both high- and low-frequency representations. % In this work, we propose a simple yet general multimodal binary segmentation framework, termed DifferSeg, to address both problems simultaneously. With the help of the differential perception fusion (DPF) module, DifferSeg employs learnable differential operators to adaptively align multimodal features and enhance their complementarity through residual fusion, effectively mitigating modality mismatch and fusion redundancy. % In addition, we design a frequency-guided decoder (FGD) that builds cross-frequency interactions and multi-path upsampling to maintain consistency between detailed high-frequency structures and semantic low-frequency representations, ensuring fine-grained boundary recovery and noise suppression. % Benefiting from these designs, DifferSeg can be easily generalized to diverse binary segmentation tasks, including both natural and medical modalities. Without bells and whistles, it consistently surpasses 67 state-of-the-art methods across 29 public datasets involving 18 downstream tasks, demonstrating superior generalization and segmentation accuracy.Code and pretrained models will be available at the Link.

Problem

• multimodal binary segmentation
• modality discrepancy
• complementarity
• frequency representation
• feature fusion

Innovation

• differential perception fusion
• frequency-guided decoder
• multimodal binary segmentation
• adaptive feature alignment
• cross-frequency interaction