Where Does Retrieval-Based Open-Ended Evaluation Fail? Automatic Taxonomy Induction from Long-Form Medical Answer Factuality Verification

2026-09-24

AI Summary

This study addresses the limitation of existing evaluation metrics for medical Retrieval-Augmented Generation (RAG) systems, which obscure specific error sources across retrieval and verification stages and lack fine-grained failure diagnostics. To this end, it constructs a two-stage error taxonomy and introduces an automated induction pipeline that operates without gold-standard evidence. The methodology employs LLM-as-Judge techniques for evidence quality annotation and reasoning error classification, combined with multi-model stress testing and an automated pattern-mining pipeline for large-scale analysis. This work makes the first systematic attempt to dissect failure modes in open-ended medical fact-checking. It demonstrates that scaling model size or applying fine-tuning cannot eliminate certain errors, revealing inherent bottlenecks within the retrieve-then-verify paradigm. All associated code and datasets are publicly released.

Abstract

Retrieval-based factuality evaluation, where LLM-generated claims are verified against evidence from authoritative medical corpora, has become the dominant paradigm for scalable hallucination detection in high-stakes clinical settings. Despite the urgency of reliable and transparent medical fact verification, most systems measure performance with aggregate metrics like F1, which obscure where and why failures occur. Existing RAG diagnostics require gold answers or annotated gold evidence, neither of which exists in this regime. We introduce two comprehensive taxonomies, grounded in a case study on the open-ended MedExpert dataset and 3 closed-ended datasets, decomposing failures into retrieval-stage errors along five quality dimensions, and verifier-reasoning errors into six consecutive steps. We adapt an automatic pattern induction pipeline using LLM-as-Judge to label evidence quality and classify verifier reasoning errors at scale, and then stress-test our findings across 4 retrieval methods and 6 frontier verifier models. Our analysis reveals that scaling model size, adding reasoning effort, expanding to authoritative web sources, and applying medical fine-tuning do not resolve these failure modes, demonstrating that they represent fundamental limitations of the retrieve-then-verify paradigm in open-ended medical settings rather than artifacts of outdated systems. We release our code and data at https://anonymous.4open.science/r/Medical_RAG_eval-4AB5 for the full reproducibility of our results.

Problem

• retrieval-based factuality evaluation
• hallucination detection
• medical RAG
• failure taxonomy
• open-ended evaluation

Innovation

• Retrieval-based factuality evaluation
• Automatic taxonomy induction
• LLM-as-Judge
• Medical hallucination detection
• RAG diagnostics