Autoregressive EHR Foundation Models with Multimodal Inputs

2026-07-24

AI Summary

This work addresses the limitation of existing autoregressive foundation models for electronic health records (EHRs), which predominantly rely on structured event codes and struggle to effectively incorporate multimodal data such as electrocardiograms, chest X-rays, and clinical notes. To overcome this, we propose a multimodal fusion framework that integrates auxiliary modalities into an autoregressive EHR model via modality-specific latent sequence compression and temporally aligned gated cross-attention mechanisms. We systematically evaluate compression strategies and pretrained encoder choices on MIMIC-IV, demonstrating that optimal compression configurations significantly outperform uncompressed cross-attention and mean pooling baselines. Furthermore, while strong pretrained encoders enhance downstream performance, realizing gains over EHR-only baselines requires carefully designed multimodal fusion architectures.

Abstract

Autoregressive foundation models trained on tokenized electronic health records (EHRs) can support zero-shot clinical prediction, yet most operate on structured event codes alone, and do not incorporate multiple modalities in a principled way. We present a framework for conditioning such models on auxiliary clinical modalities, including ECG waveforms, chest X-ray images, and clinical notes, using modality-specific latent compression and gated cross-attention with temporal alignment. We investigate two key design choices: (1) how to compress long per-modality sequences (e.g., ECG time series) before they enter the multi-modal cross-attention. This feature may be essential to reduce compute overheads and may be beneficial for generalization; (2) how the choice of pretrained encoder for each modality impacts downstream performance. Through controlled ablations on MIMIC-IV, we show that the best latent-compression configurations outperforms both uncompressed cross-attention and mean pooling. Encoder choice has a clear within-modality effect, with stronger pretrained encoders consistently outperforming weaker alternatives. We further show that merely adding auxiliary modalities does not guarantee improvement on ICU mortality prediction over an EHR-only baseline. This implies that careful design of the fusion architecture and an appropriate evaluation in the clinical context are required.

Problem

• autoregressive foundation models
• multimodal EHR
• clinical prediction
• modality fusion
• electronic health records

Innovation

• multimodal fusion
• latent compression
• gated cross-attention
• temporal alignment
• EHR foundation models