Multimodal Integration of Longitudinal Noninvasive Diagnostics for Survival Prediction in Immunotherapy Using Deep Learning

2024-11-27

JAMIA Journal of the American Medical Informatics Association

AI Summary

Predicting immunotherapy response remains challenging due to limited understanding of dynamic treatment responses in advanced-stage patients. To address this, we propose MMTSimTA—a novel multimodal temporal attention architecture that integrates longitudinal, non-invasive data (e.g., blood biomarkers, medication records, and CT-derived organ volumes) for accurate 3–12-month survival risk prediction. Our method introduces a lightweight temporal attention mechanism coupled with a dual-phase fusion strategy—integrating features at both intermediate and late stages—to enhance short-term prediction robustness. Evaluated on a pan-cancer cohort of 694 patients, MMTSimTA achieves AUCs of 0.84±0.04, 0.83±0.02, 0.82±0.02, and 0.81±0.03 for 3-, 6-, 9-, and 12-month survival prediction, respectively—significantly outperforming existing baselines. The model provides clinically interpretable, actionable insights, offering a practical AI-powered decision-support tool for personalized immunotherapy management.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Data Mining & Knowledge Management: Mining of Visual, Multimedia & Multimodal Data

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

OBJECTIVES Immunotherapies have revolutionized the landscape of cancer treatments. However, our understanding of response patterns in advanced cancers treated with immunotherapy remains limited. By leveraging routinely collected noninvasive longitudinal and multimodal data with artificial intelligence, we could unlock the potential to transform immunotherapy for cancer patients, paving the way for personalized treatment approaches. MATERIALS AND METHODS In this study, we developed a novel artificial neural network architecture, multimodal transformer-based simple temporal attention (MMTSimTA) network, building upon a combination of recent successful developments. We integrated pre- and on-treatment blood measurements, prescribed medications, and CT-based volumes of organs from a large pan-cancer cohort of 694 patients treated with immunotherapy to predict mortality at 3, 6, 9, and 12 months. Different variants of our extended MMTSimTA network were implemented and compared to baseline methods, incorporating intermediate and late fusion-based integration methods. RESULTS The strongest prognostic performance was demonstrated using a variant of the MMTSimTA model with area under the curves of 0.84 ± 0.04, 0.83 ± 0.02, 0.82 ± 0.02, 0.81 ± 0.03 for 3-, 6-, 9-, and 12-month survival prediction, respectively. DISCUSSION Our findings show that integrating noninvasive longitudinal data using our novel architecture yields an improved multimodal prognostic performance, especially in short-term survival prediction. CONCLUSION Our study demonstrates that multimodal longitudinal integration of noninvasive data using deep learning may offer a promising approach for personalized prognostication in immunotherapy-treated cancer patients.

Problem

• Predicting survival in immunotherapy-treated cancer patients
• Integrating multimodal longitudinal noninvasive diagnostic data
• Improving prognostic accuracy using deep learning

Innovation

• Multimodal transformer-based neural network
• Integration of noninvasive longitudinal data
• Deep learning for survival prediction