Mix-modal Federated Learning for MRI Image Segmentation

2025-09-02

AI Summary

In decentralized medical settings, existing centralized multimodal segmentation methods fail due to severe heterogeneity across hospitals in both MRI modality composition and data distribution—termed *dual heterogeneity* (modality and data). To address this, we propose MixMFL, the first federated learning paradigm for decentralized hybrid-modality MRI segmentation. Its core innovations are: (i) a *modality-decoupled encoder* that disentangles modality-specific and shared latent representations; and (ii) a *dynamic modality prototype memory* enabling adaptive inter-client modality fusion and missing-modality compensation during federated aggregation. Extensive experiments on two public MRI datasets demonstrate that MixMFL significantly outperforms state-of-the-art federated segmentation methods, achieving superior robustness—particularly on clients with incomplete modality sets—while maintaining high segmentation accuracy under realistic cross-hospital data heterogeneity.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Systems and Infrastructure for Web, Mobile and WoT: Decentralized Web and Fediverse systems

Abstract

Magnetic resonance imaging (MRI) image segmentation is crucial in diagnosing and treating many diseases, such as brain tumors. Existing MRI image segmentation methods mainly fall into a centralized multimodal paradigm, which is inapplicable in engineering non-centralized mix-modal medical scenarios. In this situation, each distributed client (hospital) processes multiple mixed MRI modalities, and the modality set and image data for each client are diverse, suffering from extensive client-wise modality heterogeneity and data heterogeneity. In this paper, we first formulate non-centralized mix-modal MRI image segmentation as a new paradigm for federated learning (FL) that involves multiple modalities, called mix-modal federated learning (MixMFL). It distinguishes from existing multimodal federating learning (MulMFL) and cross-modal federating learning (CroMFL) paradigms. Then, we proposed a novel modality decoupling and memorizing mix-modal federated learning framework (MDM-MixMFL) for MRI image segmentation, which is characterized by a modality decoupling strategy and a modality memorizing mechanism. Specifically, the modality decoupling strategy disentangles each modality into modality-tailored and modality-shared information. During mix-modal federated updating, corresponding modality encoders undergo tailored and shared updating, respectively. It facilitates stable and adaptive federating aggregation of heterogeneous data and modalities from distributed clients. Besides, the modality memorizing mechanism stores client-shared modality prototypes dynamically refreshed from every modality-tailored encoder to compensate for incomplete modalities in each local client. It further benefits modality aggregation and fusion processes during mixmodal federated learning. Extensive experiments on two public datasets for MRI image segmentation demonstrate the effectiveness and superiority of our methods.

Problem

• Addresses MRI segmentation in decentralized mix-modal medical settings
• Handles client-wise modality and data heterogeneity in federated learning
• Proposes modality decoupling and memorizing for incomplete modality compensation

Innovation

• Modality decoupling strategy for tailored and shared information
• Modality memorizing mechanism storing client-shared prototypes
• Dynamic prototype refreshment compensating incomplete local modalities