EHRBench: An Automated and Reliable EHR-based Benchmark for Clinical Decision Making with LLMs

2026-05-28

AI Summary

This study addresses the current lack of scalable and reliable benchmarks for evaluating large clinical decision-making models based on real-world electronic health records (EHRs). The authors propose an innovative paradigm that synergistically integrates EHR data, large language models, and medical knowledge bases to automatically construct structured clinical question-answer pairs covering diagnosis, treatment, and prognosis. By leveraging specialized prompt templates, deterministic instantiation, and systematic knowledge validation, they generate a high-quality evaluation dataset comprising 960,000 samples. Using this benchmark, the study comprehensively evaluates over thirty prominent large language models, offering the first systematic analysis of their capabilities and robustness limitations in clinical reasoning tasks.

Abstract

Clinical decision-making (CDM) is central to real-world clinical workflows, where clinicians infer diagnoses, select treatments, or anticipate future health outcomes under incomplete evidence. LLMs are increasingly used to support these decisions due to strong language capabilities, broad biomedical knowledge, and efficiency, yet the reliability of LLMs on real-world clinical decision tasks remains insufficiently understood. To evaluate CDM models, especially LLM-based models, an ideal and practical medical decision benchmark should be constructed via an automated yet reliable pipeline to ensure both scale and quality. Moreover, the grounding of a CDM benchmark in real patient EHRs can better support evaluation on practical CDM tasks that require substantive biomedical knowledge and clinical inference. To fill the gaps, we introduce EHRBench, an automated and reliable EHR-grounded benchmark for evaluating LLM-based clinical decision-making at scale. To ensure scalability and reliability, EHRBench is constructed through an EHR-LLM-KB(knowledge-base) interaction pipeline. For efficiency, we use a specialized LLM to automatically convert encounter-level EHR trajectories into structured templates and deterministically instantiate the templates into QA items. In parallel, we apply systematic KB-based verification and enrichment to filter hallucinated or ambiguous relations and to improve reliability. Using this pipeline, we construct nearly 1M (960,067) QA items spanning three core inference-required clinical decision tasks: diagnosis, treatment, and prognosis. We benchmark more than 30 representative LLMs on EHRBench and provide detailed analyses of performance and robustness. The results show consistent capability trends across settings, further validating the reliability of EHRBench and highlighting actionable gaps toward clinically reliable LLM systems.

Problem

• clinical decision-making
• large language models
• electronic health records
• benchmark
• reliability

Innovation

• EHRBench
• clinical decision making
• large language models
• knowledge-base verification
• automated benchmarking