Pretraining EHR Foundation Models with Patient-Aware Sampling

2026-07-24

AI Summary

This work addresses a critical limitation in conventional autoregressive pretraining for electronic health records (EHRs), where trajectories from multiple patients are concatenated into a single global sequence, leading to training windows that intermingle data across individuals and allowing patients with longer records to dominate the optimization process—introducing significant bias. The study presents the first systematic investigation into the impact of sequence construction strategies in EHR foundation models and proposes Patient Sampling, a novel approach that constructs sequences on a per-patient basis during pretraining. By employing controlled random sampling, this method ensures balanced contribution of training signals across all patients. Evaluated within an autoregressive language modeling framework on MIMIC-IV v2.2 and v3.1, Patient Sampling consistently outperforms the standard Global Stream baseline across multiple downstream clinical tasks, achieving notable improvements in both Macro AUROC and AUPRC.

Abstract

Autoregressive foundation models for electronic health records (EHRs) typically inherit pretraining methods from language modeling, where patient trajectories are concatenated into a single token stream and windows are sampled from that stream. In EHR data, this choice is consequential: windows may mix multiple patients, and patients with longer records contribute more optimization updates, potentially introducing bias. We propose Patient Sampling, a pretraining sequence-construction method that allows us to control how training signal is distributed across patients. We compare this method to the standard approach, which we refer to as Global Stream. We show that stochastic Patient Sampling with controllable weighting improves performance on real-world EHR data. Across downstream clinical tasks on MIMIC-IV v2.2 and v3.1, Patient Sampling improves Macro AUROC and AUPRC over the Global Stream baseline. These results identify training and validation sequence construction as important and underexplored design choices for autoregressive EHR foundation models.

Problem

• Electronic Health Records
• Foundation Models
• Pretraining Bias
• Patient Sampling
• Autoregressive Modeling

Innovation

• Patient Sampling
• EHR foundation models
• autoregressive modeling
• sequence construction
• clinical prediction