Multi-Modal Mamba Modeling for Survival Prediction (M4Survive): Adapting Joint Foundation Model Representations

2025-03-13

AI Summary

To address insufficient multimodal information fusion in cancer survival prediction, this paper proposes a lightweight dynamic cross-modal fusion method built upon the Mamba architecture. Our approach introduces learnable adapters to dynamically align and jointly represent radiological and pathological images within heterogeneous foundation model embedding spaces (e.g., MedImageInsight, BiomedCLIP). By integrating Mamba’s efficient sequence modeling capability with Cox regression, we construct an end-to-end framework for survival risk prediction. Evaluated on multiple public benchmarks, our method consistently outperforms unimodal baselines and conventional static multimodal approaches, achieving up to a 4.2% improvement in concordance index (C-index). It delivers both high predictive accuracy and low computational overhead, offering a novel paradigm for precision oncology prognosis.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization

Abstract

Accurate survival prediction in oncology requires integrating diverse imaging modalities to capture the complex interplay of tumor biology. Traditional single-modality approaches often fail to leverage the complementary insights provided by radiological and pathological assessments. In this work, we introduce M4Survive (Multi-Modal Mamba Modeling for Survival Prediction), a novel framework that learns joint foundation model representations using efficient adapter networks. Our approach dynamically fuses heterogeneous embeddings from a foundation model repository (e.g., MedImageInsight, BiomedCLIP, Prov-GigaPath, UNI2-h), creating a correlated latent space optimized for survival risk estimation. By leveraging Mamba-based adapters, M4Survive enables efficient multi-modal learning while preserving computational efficiency. Experimental evaluations on benchmark datasets demonstrate that our approach outperforms both unimodal and traditional static multi-modal baselines in survival prediction accuracy. This work underscores the potential of foundation model-driven multi-modal fusion in advancing precision oncology and predictive analytics.

Problem

• Integrates diverse imaging modalities for accurate oncology survival prediction.
• Overcomes limitations of single-modality approaches in tumor biology analysis.
• Proposes M4Survive for efficient multi-modal learning and survival risk estimation.

Innovation

• Uses Mamba-based adapters for efficient learning
• Integrates diverse imaging modalities dynamically
• Creates optimized latent space for survival prediction