EHRCon: Dataset for Checking Consistency between Unstructured Notes and Structured Tables in Electronic Health Records

2024-06-24

arXiv.org

AI Summary

Electronic health records (EHRs) frequently exhibit semantic inconsistencies between structured tabular data and unstructured clinical notes, posing risks to clinical decision-making safety. To address this challenge, we introduce EHRCon—the first benchmark dataset for cross-modal consistency verification in EHRs—comprising 105 clinical notes and 4,101 annotated entities, fully compatible with both MIMIC-III and OMOP Common Data Model standards, and rigorously validated by domain experts. We formalize a novel paradigm for EHR cross-modal consistency validation and propose CheckEHR, an eight-stage large language model–driven framework supporting zero-shot and few-shot inference. Extensive experiments demonstrate that CheckEHR significantly outperforms existing baselines. EHRCon exhibits high annotation confidence and strong cross-system reproducibility. Both the dataset and implementation code are publicly released to foster reproducible research.

Technology Category

• Computer Vision: Multi-modal Vision
• Knowledge Representation and Reasoning: Common-Sense Reasoning
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Security and Privacy: Data transparency and provenance

Abstract

Electronic Health Records (EHRs) are integral for storing comprehensive patient medical records, combining structured data (e.g., medications) with detailed clinical notes (e.g., physician notes). These elements are essential for straightforward data retrieval and provide deep, contextual insights into patient care. However, they often suffer from discrepancies due to unintuitive EHR system designs and human errors, posing serious risks to patient safety. To address this, we developed EHRCon, a new dataset and task specifically designed to ensure data consistency between structured tables and unstructured notes in EHRs. EHRCon was crafted in collaboration with healthcare professionals using the MIMIC-III EHR dataset, and includes manual annotations of 4,101 entities across 105 clinical notes checked against database entries for consistency. EHRCon has two versions, one using the original MIMIC-III schema, and another using the OMOP CDM schema, in order to increase its applicability and generalizability. Furthermore, leveraging the capabilities of large language models, we introduce CheckEHR, a novel framework for verifying the consistency between clinical notes and database tables. CheckEHR utilizes an eight-stage process and shows promising results in both few-shot and zero-shot settings. The code is available at https://github.com/dustn1259/EHRCon.

Problem

• Electronic Health Records
• Inconsistency
• Data Verification

Innovation

• EHRCon
• CheckEHR
• MIMIC-III