Completed Feature Disentanglement Learning for Multimodal MRIs Analysis

2024-07-06

IEEE journal of biomedical and health informatics

AI Summary

Existing feature disentanglement (FD)-based multimodal MRI methods suffer from loss of cross-modal shared information when handling ≥3 modalities and lack explicit modeling of relationships among disentangled features during fusion. To address these limitations, we propose a Complete Feature Disentanglement (CFD) strategy and a Dynamic Mixture-of-Experts Fusion (DMF) module. CFD introduces the novel concept of “modality-partially shared features,” enabling full-order disentanglement that preserves both modality-specific and hierarchically shared representations. The DMF module employs a dynamic gating mechanism to jointly model local–global feature interactions, thereby enhancing both interpretability and fusion accuracy. Evaluated on three multimodal MRI classification tasks, our approach consistently outperforms state-of-the-art methods, demonstrating the effectiveness of restoring shared-information integrity and dynamically modeling inter-feature relationships.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Multimodal MRIs play a crucial role in clinical diagnosis and treatment. Feature disentanglement (FD)-based methods, aiming at learning superior feature representations for multimodal data analysis, have achieved significant success in multimodal learning (MML). Typically, existing FD-based methods separate multimodal data into modality-shared and modality-specific features, and employ concatenation or attention mechanisms to integrate these features. However, our preliminary experiments indicate that these methods could lead to a loss of shared information among subsets of modalities when the inputs contain more than two modalities, and such information is critical for prediction accuracy. Furthermore, these methods do not adequately interpret the relationships between the decoupled features at the fusion stage. To address these limitations, we propose a novel Complete Feature Disentanglement (CFD) strategy that recovers the lost information during feature decoupling. Specifically, the CFD strategy not only identifies modality-shared and modality-specific features, but also decouples shared features among subsets of multimodal inputs, termed as modality-partial-shared features. We further introduce a new Dynamic Mixture-of-Experts Fusion (DMF) module that dynamically integrates these decoupled features, by explicitly learning the local-global relationships among the features. The effectiveness of our approach is validated through classification tasks on three multimodal MRI datasets. Extensive experimental results demonstrate that our approach outperforms other state-of-the-art MML methods with obvious margins, showcasing its superior performance.

Problem

• Loss of shared information in multimodal MRIs
• Inadequate feature relationship interpretation
• Improved feature disentanglement and fusion strategies

Innovation

• Complete Feature Disentanglement strategy
• Dynamic Mixture-of-Experts Fusion module
• Recovers lost shared information