DC-Seg: Disentangled Contrastive Learning for Brain Tumor Segmentation with Missing Modalities

2025-05-17

AI Summary

To address insufficient robustness in multimodal brain tumor segmentation caused by missing imaging modalities, this paper proposes an anatomy-modal representation disentanglement framework. Methodologically, it introduces, for the first time, a synergistic anatomy- and modality-aware contrastive learning mechanism, coupled with a segmentation-guided regularizer to explicitly decouple cross-modal anatomical structure representations from modality-specific features. The framework incorporates multimodal latent space decomposition, a hybrid Transformer-CNN encoder, and segmentation consistency regularization. Evaluated on BraTS 2020 and a private WMH dataset, the method achieves Dice score improvements of 3.2–5.8% under single-modality input, significantly outperforming state-of-the-art approaches while demonstrating strong generalizability. The source code is publicly available.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Accurate segmentation of brain images typically requires the integration of complementary information from multiple image modalities. However, clinical data for all modalities may not be available for every patient, creating a significant challenge. To address this, previous studies encode multiple modalities into a shared latent space. While somewhat effective, it remains suboptimal, as each modality contains distinct and valuable information. In this study, we propose DC-Seg (Disentangled Contrastive Learning for Segmentation), a new method that explicitly disentangles images into modality-invariant anatomical representation and modality-specific representation, by using anatomical contrastive learning and modality contrastive learning respectively. This solution improves the separation of anatomical and modality-specific features by considering the modality gaps, leading to more robust representations. Furthermore, we introduce a segmentation-based regularizer that enhances the model's robustness to missing modalities. Extensive experiments on the BraTS 2020 and a private white matter hyperintensity(WMH) segmentation dataset demonstrate that DC-Seg outperforms state-of-the-art methods in handling incomplete multimodal brain tumor segmentation tasks with varying missing modalities, while also demonstrate strong generalizability in WMH segmentation. The code is available at https://github.com/CuCl-2/DC-Seg.

Problem

• Segments brain tumors with missing imaging modalities
• Disentangles modality-invariant and modality-specific image features
• Improves robustness in incomplete multimodal segmentation tasks

Innovation

• Disentangles images into modality-invariant and specific representations
• Uses anatomical and modality contrastive learning
• Introduces segmentation-based regularizer for missing modalities