EarlyDx: An Admission-Anchored Benchmark for Open-Ended Generation of Evidence-Supported ED-Encounter Diagnoses

2026-07-30

AI Summary

This study addresses the challenge of early diagnosis in emergency settings, where clinicians must make rapid decisions under limited information—a scenario poorly captured by existing benchmarks that rely on closed label sets, ignore free-text clinical notes, and use discharge diagnoses as supervision, which misaligns with real-time clinical workflows. To bridge this gap, the authors propose EarlyDx, the first open-ended generative diagnostic benchmark tailored to the emergency admission context. Built on 154,834 emergency encounters from MIMIC-IV, EarlyDx uses only data available at admission and employs in-emergency-department diagnoses as supervision. It innovatively leverages a large language model as an auditor to grade evidence support for free-text diagnoses, integrating structured and unstructured data into a multimodal benchmark with a hierarchical evaluation protocol. Experiments reveal that current models exhibit limited performance on reasoning-intensive diagnostic tasks, recovering only 3–31% of diagnoses in zero-shot settings and achieving 56% recall after fine-tuning, yet still failing to meet clinical requirements for sensitivity–precision balance in time-sensitive conditions.

Abstract

Clinical diagnosis at hospital admission must be made rapidly from limited, incomplete evidence. Existing diagnosis-prediction benchmarks are poorly suited to this setting: they restrict prediction to closed code sets, exclude free-text notes, and supervise with discharge diagnoses that incorporate the full inpatient course. We introduce EarlyDx, a large-scale benchmark for open-ended early diagnosis, built from 154,834 emergency department encounters in MIMIC-IV. Each encounter is restricted to records available at admission time $t_0$ and supervised by the diagnoses recorded during the ED encounter rather than at discharge. An LLM auditor further verifies every free-text label as supported, partially supported, or unsupported by that evidence; the primary evaluation scores only fully supported labels. Under a semantic LLM-as-judge protocol, no evaluated system --- frontier general, medical-specialized, or in-domain post-trained --- synthesizes admission-time evidence reliably. Zero-shot models score largely by extraction, recovering only 3-31% of diagnoses that must be inferred rather than read from the record; post-training raises inference-dependent recall to 56%, but a sizeable margin remains, and on time-critical conditions no system attains a clinician's balance of sensitivity and precision. We release the full construction and evaluation pipeline at here.

Problem

• early diagnosis
• emergency department
• open-ended generation
• evidence-supported
• admission-time prediction

Innovation

• EarlyDx
• open-ended diagnosis generation
• admission-anchored benchmark
• evidence-supported labeling
• LLM-as-judge evaluation