Benchmarking Foundation Models with Multimodal Public Electronic Health Records

2025-07-20

AI Summary

This study addresses critical challenges—performance degradation, fairness disparities, and lack of interpretability—in deploying multimodal electronic health record (EHR) foundation models clinically. To this end, we establish the first standardized multimodal EHR benchmark built on MIMIC-IV. We propose a unified preprocessing pipeline for heterogeneous clinical data and conduct the first systematic evaluation of eight state-of-the-art unimodal and multimodal foundation models—including both domain-specific and general-purpose architectures—on predictive tasks. Experimental results demonstrate that multimodal fusion consistently and significantly improves predictive performance across diverse clinical outcomes, without exacerbating inter-group bias; furthermore, model behavior exhibits inherent interpretability. All code, preprocessing protocols, and evaluation frameworks are fully open-sourced to enable reproducible, verifiable, and trustworthy medical AI research.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Foundation models have emerged as a powerful approach for processing electronic health records (EHRs), offering flexibility to handle diverse medical data modalities. In this study, we present a comprehensive benchmark that evaluates the performance, fairness, and interpretability of foundation models, both as unimodal encoders and as multimodal learners, using the publicly available MIMIC-IV database. To support consistent and reproducible evaluation, we developed a standardized data processing pipeline that harmonizes heterogeneous clinical records into an analysis-ready format. We systematically compared eight foundation models, encompassing both unimodal and multimodal models, as well as domain-specific and general-purpose variants. Our findings demonstrate that incorporating multiple data modalities leads to consistent improvements in predictive performance without introducing additional bias. Through this benchmark, we aim to support the development of effective and trustworthy multimodal artificial intelligence (AI) systems for real-world clinical applications. Our code is available at https://github.com/nliulab/MIMIC-Multimodal.

Problem

• Evaluating performance, fairness, interpretability of foundation models in EHRs
• Standardizing data processing for heterogeneous clinical records analysis
• Comparing multimodal and unimodal models for clinical AI applications

Innovation

• Standardized pipeline for harmonizing clinical records
• Benchmarking eight foundation models systematically
• Multimodal models improve predictive performance consistently