🤖 AI Summary
This work proposes the first risk-sensitive hallucination evaluation framework for medical question-answering models, addressing a critical gap in existing assessments that focus narrowly on factual accuracy while overlooking variations in clinical risk. By identifying risk-laden language—such as treatment directives, contraindications, urgent warnings, and high-risk medications—and integrating relevance-based metrics, the framework quantifies the potential harm arising from user adherence to hallucinated responses. It introduces a clinical risk dimension to distinguish between low- and high-risk hallucinations, revealing safety disparities invisible to conventional evaluation metrics. Experiments on three leading instruction-tuned large language models demonstrate that models exhibiting similar surface-level behaviors can possess markedly different risk profiles, underscoring the necessity and effectiveness of this approach for comprehensive safety assessment in medical AI systems.
📝 Abstract
Large language models are increasingly being used in patient-facing medical question answering, where hallucinated outputs can vary widely in potential harm. However, existing hallucination standards and evaluation metrics focus primarily on factual correctness, treating all errors as equally severe. This obscures clinically relevant failure modes, particularly when models generate unsupported but actionable medical language. We propose a risk-sensitive evaluation framework that quantifies hallucinations through the presence of risk-bearing language, including treatment directives, contraindications, urgency cues, and mentions of high-risk medications. Rather than assessing clinical correctness, our approach evaluates the potential impact of hallucinated content if acted upon. We further combine risk scoring with a relevance measure to identify high-risk, low-grounding failures. We apply this framework to three instruction-tuned language models using controlled patient-facing prompts designed as safety stress tests. Our results show that models with similar surface-level behavior exhibit substantially different risk profiles and that standard evaluation metrics fail to capture these distinctions. These findings highlight the importance of incorporating risk sensitivity into hallucination evaluation and suggest that evaluation validity is critically dependent on task and prompt design.