🤖 AI Summary
Current evaluation methods for clinical document generation often misclassify valid medical reasoning as hallucination, leading to an underestimation of large language model performance. This work proposes a clinically grounded evaluation framework that redefines “hallucination” in SOAP notes—not by literal fidelity but by clinical plausibility—through calibrated prompt engineering, a retrieval-augmented mechanism supported by medical ontologies such as SNOMED CT, and a reasoning-aware assessment pipeline. The framework effectively distinguishes genuine hallucinations from legitimate clinical abstractions, including terminology mapping, diagnostic inference, and guideline-concordant care planning. Experimental results demonstrate a significant reduction in average hallucination rates from 35% to 9%, with the remaining cases predominantly involving actual safety concerns, thereby validating the framework’s efficacy and clinical relevance.
📝 Abstract
Evaluating large language models (LLMs) for clinical documentation tasks such as SOAP note generation remains challenging. Unlike standard summarization, these tasks require clinical abstraction, normalization of colloquial language, and medically grounded inference. However, prevailing evaluation methods including automated metrics and LLM as judge frameworks rely on lexical faithfulness, often labeling any information not explicitly present in the transcript as hallucination.
We show that such approaches systematically misclassify clinically valid outputs as errors, inflating hallucination rates and distorting model assessment. Our analysis reveals that many flagged hallucinations correspond to legitimate clinical transformations, including synonym mapping, abstraction of examination findings, diagnostic inference, and guideline consistent care planning.
By aligning evaluation criteria with clinical reasoning through calibrated prompting and retrieval grounded in medical ontologies we observe a significant shift in outcomes. Under a lexical evaluation regime, the mean hallucination rate is 35%, heavily penalizing valid reasoning. With inference aware evaluation, this drops to 9%, with remaining cases reflecting genuine safety concerns. These findings suggest that current evaluation practices over penalize valid clinical reasoning and may measure artifacts of evaluation design rather than true errors, underscoring the need for clinically informed evaluation in high context domains like medicine.