đ€ AI Summary
Prior clinical prediction studies for heart failure lack systematic ablation analyses of preprocessing pipelines and model architectures. Method: Leveraging a large-scale Swedish EHR cohort, we conduct the first comprehensive ablation study in clinical forecastingâsystematically evaluating input tokenization strategies, temporal preprocessing techniques, and architectural choices across six sequential models (including Transformer, Llama-enhanced Transformer++, and Mamba), while integrating multimodal time-series data (diagnoses, vital signs, laboratory tests, medications, and procedures). Results: Llama-enhanced Transformer++ and Mamba achieve superior performance over standard Transformerâdespite substantially fewer parametersâdemonstrating higher accuracy, better calibration, and greater robustness. Notably, both models surpass the large Transformerâs performance using only 75% of the training data. This work establishes a reproducible benchmark for EHR sequence modeling and provides empirically grounded design principles for lightweight, efficient clinical prediction models.
đ Abstract
Transformers have defined the state-of-the-art for clinical prediction tasks involving electronic health records (EHRs). The recently introduced Mamba architecture outperformed an advanced Transformer (Transformer++) based on Llama in handling long context lengths, while using fewer model parameters. Despite the impressive performance of these architectures, a systematic approach to empirically analyze model performance and efficiency under various settings is not well established in the medical domain. The performances of six sequence models were investigated across three architecture classes (Transformers, Transformers++, Mambas) in a large Swedish heart failure (HF) cohort (N = 42820), providing a clinically relevant case study. Patient data included diagnoses, vital signs, laboratories, medications and procedures extracted from in-hospital EHRs. The models were evaluated on three one-year prediction tasks: clinical instability (a readmission phenotype) after initial HF hospitalization, mortality after initial HF hospitalization and mortality after latest hospitalization. Ablations account for modifications of the EHR-based input patient sequence, architectural model configurations, and temporal preprocessing techniques for data collection. Llama achieves the highest predictive discrimination, best calibration, and showed robustness across all tasks, followed by Mambas. Both architectures demonstrate efficient representation learning, with tiny configurations surpassing other large-scaled Transformers. At equal model size, Llama and Mambas achieve superior performance using 25% less training data. This paper presents a first ablation study with systematic design choices for input tokenization, model configuration and temporal data preprocessing. Future model development in clinical prediction tasks using EHRs could build upon this study's recommendation as a starting point.