A Dataset for Addressing Patient's Information Needs related to Clinical Course of Hospitalization

2025-06-04

AI Summary

Existing EHR question-answering (QA) research lacks evaluation benchmarks aligned with real-world clinical information needs of hospitalized patients. Method: We introduce ArchEHR-QA—the first patient-centered EHR QA dataset—comprising 134 real ICU and emergency department cases, each annotated with patient-formulated questions, clinician-provided answers, clinical notes, and fine-grained sentence-level relevance labels. We systematically characterize inpatient clinical information needs and propose a multidimensional evaluation framework jointly assessing factual accuracy and relevance. Contribution/Results: Leveraging Llama-4, Llama-3, and Mixtral, we benchmark three prompting strategies and find “answer-first” prompting optimal, with Llama-4 achieving the highest performance. Error analysis identifies missing critical evidence and hallucination as primary failure modes. ArchEHR-QA establishes a reproducible, patient-oriented benchmark for clinical QA research, enabling rigorous evaluation of model fidelity to real-world clinical decision-support requirements.

Technology Category

• Natural Language Processing: Question Answering
• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Patients have distinct information needs about their hospitalization that can be addressed using clinical evidence from electronic health records (EHRs). While artificial intelligence (AI) systems show promise in meeting these needs, robust datasets are needed to evaluate the factual accuracy and relevance of AI-generated responses. To our knowledge, no existing dataset captures patient information needs in the context of their EHRs. We introduce ArchEHR-QA, an expert-annotated dataset based on real-world patient cases from intensive care unit and emergency department settings. The cases comprise questions posed by patients to public health forums, clinician-interpreted counterparts, relevant clinical note excerpts with sentence-level relevance annotations, and clinician-authored answers. To establish benchmarks for grounded EHR question answering (QA), we evaluated three open-weight large language models (LLMs)--Llama 4, Llama 3, and Mixtral--across three prompting strategies: generating (1) answers with citations to clinical note sentences, (2) answers before citations, and (3) answers from filtered citations. We assessed performance on two dimensions: Factuality (overlap between cited note sentences and ground truth) and Relevance (textual and semantic similarity between system and reference answers). The final dataset contains 134 patient cases. The answer-first prompting approach consistently performed best, with Llama 4 achieving the highest scores. Manual error analysis supported these findings and revealed common issues such as omitted key clinical evidence and contradictory or hallucinated content. Overall, ArchEHR-QA provides a strong benchmark for developing and evaluating patient-centered EHR QA systems, underscoring the need for further progress toward generating factual and relevant responses in clinical contexts.

Problem

• Lack of dataset for patient information needs in EHRs
• Need to evaluate AI-generated responses for accuracy and relevance
• Developing benchmarks for grounded EHR question answering

Innovation

• Expert-annotated dataset ArchEHR-QA for EHR QA
• Evaluated LLMs with three prompting strategies
• Assessed performance on Factuality and Relevance