🤖 AI Summary
To address the temporal heterogeneity, sparsity, and modality granularity discrepancies inherent in multimodal clinical time-series data—such as electronic health records (EHR) and electrocardiograms (ECG)—this paper proposes a hierarchical multimodal temporal fusion model. Methodologically, it introduces a sparse time-series encoder, a hierarchical temporal fusion module, and a dual-modal attention mechanism, augmented by modality-specific pretrained encoders and a shared latent-space feature alignment strategy to enable dynamic cross-modal interaction and unified multi-granularity temporal representation learning. Evaluated on MIMIC-IV and MIMIC-IV-ECG, the model achieves state-of-the-art performance: AUROC = 0.947 for 90-day cardiovascular event prediction, AUROC = 0.901 for in-hospital mortality prediction, and MAE = 2.31 hours for ICU length-of-stay regression. It demonstrates strong generalizability and scalability across diverse clinical forecasting tasks.
📝 Abstract
The inherent multimodality and heterogeneous temporal structures of medical data pose significant challenges for modeling. We propose MedM2T, a time-aware multimodal framework designed to address these complexities. MedM2T integrates: (i) Sparse Time Series Encoder to flexibly handle irregular and sparse time series, (ii) Hierarchical Time-Aware Fusion to capture both micro- and macro-temporal patterns from multiple dense time series, such as ECGs, and (iii) Bi-Modal Attention to extract cross-modal interactions, which can be extended to any number of modalities. To mitigate granularity gaps between modalities, MedM2T uses modality-specific pre-trained encoders and aligns resulting features within a shared encoder. We evaluated MedM2T on MIMIC-IV and MIMIC-IV-ECG datasets for three tasks that encompass chronic and acute disease dynamics: 90-day cardiovascular disease (CVD) prediction, in-hospital mortality prediction, and ICU length-of-stay (LOS) regression. MedM2T outperformed state-of-the-art multimodal learning frameworks and existing time series models, achieving an AUROC of 0.947 and an AUPRC of 0.706 for CVD prediction; an AUROC of 0.901 and an AUPRC of 0.558 for mortality prediction; and Mean Absolute Error (MAE) of 2.31 for LOS regression. These results highlight the robustness and broad applicability of MedM2T, positioning it as a promising tool in clinical prediction. We provide the implementation of MedM2T at https://github.com/DHLab-TSENG/MedM2T.