Training Large Language Models to Predict Clinical Events

2026-05-12

AI Summary

This work proposes a novel method for automatically extracting supervision signals from longitudinal clinical notes to enable multiclass clinical event prediction. Leveraging the Foresight Learning framework, the approach transforms sequential clinical narratives into predictive instances comprising historical context, natural language queries, and verification labels derived from subsequent documentation—eliminating the need for handcrafted structured features or task-specific classifiers. Evaluated on the MIMIC-III dataset with lightweight LoRA-based fine-tuning, the model substantially outperforms prompt-learning baselines, achieving an expected calibration error of 0.0398 and a Brier score of 0.145, slightly surpassing GPT-5 point estimates. This study presents the first end-to-end system capable of generating reusable supervisory signals for clinical prediction directly from unstructured clinical text.

Abstract

Longitudinal clinical notes contain rich evidence of how patients evolve over time, but converting this signal into training supervision for clinical prediction remains challenging. We extend Foresight Learning to clinical prediction by converting time-ordered MIMIC-III notes into examples consisting of past patient context, a natural-language question about a possible future event, and a label resolved from later documentation. This process yields 6,900 prediction examples from 702 admissions across medications, procedures, organ support, microbiology, and mortality. A small LoRA adapter trained on these examples improves over the prompted base model, reducing expected calibration error from 0.1269 to 0.0398 and Brier score from 0.199 to 0.145, while slightly outperforming GPT-5 point estimates on held-out questions. The approach enables reusable clinical prediction supervision from longitudinal notes without hand-engineered structured features or endpoint-specific classifiers.

Problem

• clinical prediction
• longitudinal clinical notes
• supervision signal
• large language models
• MIMIC-III

Innovation

• Foresight Learning
• clinical prediction
• longitudinal clinical notes
• LoRA adaptation
• natural-language supervision