clinical expert evaluation

This paper identifies a “retrieval-reasoning disconnection” problem in large language models (LLMs) deployed in clinical settings: even when authoritative clinical guidelines—such as Written Exposure Therapy (WET)—are accurately retrieved, LLMs frequently generate reasoning outputs that violate structured protocol requirements. To address this, we introduce the first RAG evaluation framework specifically designed for clinical protocol adherence, quantifying reasoning quality along three dimensions—accuracy, consistency, and fidelity. Leveraging an expert-validated WET question set and corresponding guideline documents, we conduct a systematic empirical analysis across leading LLMs. Results show that while current RAG systems constrain output format, critical clinical reasoning errors persist at a high rate of 32%. Our key contribution is the formal integration of reasoning processes—alongside retrieval—into rigorous evaluation. We further provide a reproducible benchmark and actionable improvement pathways for structured medical decision-making.

Current large language models lack structured evaluation of core therapeutic principles in mental health conversations, compromising clinical appropriateness. To address this gap, this work introduces FAITH-M—the first expert-annotated benchmark grounded in established therapeutic principles—and CARE, a multi-stage evaluation framework that enables precise assessment of AI therapist responses through fine-grained ordinal scoring, context-aware analysis, contrastive example retrieval, and chain-of-thought knowledge distillation. Using Qwen3 as the backbone model, CARE achieves an F1 score of 63.34, representing a 64.26% improvement over baseline methods, and demonstrates strong robustness across diverse datasets and expert evaluations.

Clinical risk-scoring benchmarks such as MedCalc-Bench suffer from the entrenchment of historically erroneous models as de facto “gold standards,” particularly problematic when used as reward signals in reinforcement learning, where such biases are amplified. Method: We propose a “living documentation” benchmark paradigm featuring physician-in-the-loop, low-burden dynamic curation: GRPO-based RL fine-tuning of Qwen3-8B, multi-stage agentic verification, LLM-driven logical consistency checking, and clinical-knowledge-guided controversy identification—enabling scalable, trustworthy re-annotation. Contribution/Results: Experiments uncover substantial label noise; post-correction, model accuracy improves by 8.7 percentage points. This work is the first systematic demonstration that safety-critical domain benchmarks require continuous governance, establishing a novel human–AI collaborative paradigm for dynamic benchmark auditing.

Current large language models (LLMs) lack rigorous, clinically grounded evaluation of reasoning interpretability in complex medical decision-making. Method: We introduce JAMA Clinical Challenge and Medbullets—two high-difficulty, multiple-choice clinical benchmarks featuring authoritative, fine-grained expert explanations—the first such resources designed for real-world clinical scenarios. Evaluation employs zero-shot and few-shot prompting, automated explanation quality scoring, dual-blinded clinical expert assessment (Cohen’s κ = 0.82), and comparative analysis. Contribution/Results: Seven state-of-the-art LLMs exhibit substantially lower performance on these benchmarks than on conventional exam-style benchmarks. Their generated explanations frequently contain logical gaps and factual hallucinations, exposing critical deficiencies in clinical-grade causal reasoning and domain-specific knowledge integration—highlighting a fundamental gap between current LLM capabilities and safe, interpretable clinical deployment.

Despite widespread adoption of domain-specific fine-tuning for biomedical large language models (LLMs), there remains a lack of systematic empirical evidence on its actual impact on clinical capabilities. Method: We conduct a comprehensive evaluation of 25 state-of-the-art LLMs—including both general-purpose and biomedical-fine-tuned variants—across six standardized clinical tasks, using CLUE, a reproducible, open-source medical evaluation framework (with all code and data publicly released). Contribution/Results: Our study is the first to empirically demonstrate that most biomedical-fine-tuned models underperform general-purpose models in critical clinical competencies—including hallucination suppression, ICD-10 coding accuracy, and instruction following. Notably, Llama-3.1-70B-Instruct surpasses specialized biomedical models across multiple tasks, revealing inherent trade-offs in domain adaptation. These findings challenge the prevailing assumption that biomedical fine-tuning inherently enhances clinical performance, establishing a rigorous empirical benchmark and offering methodological insights for LLM deployment in healthcare.

Large language models (LLMs) demonstrate strong performance in static medical question-answering tasks but exhibit significantly degraded capabilities in clinical diagnostic scenarios requiring multi-turn evidence gathering. To address this gap, this work introduces a standardized patient simulator grounded in Objective Structured Clinical Examination (OSCE) principles and establishes a reproducible benchmark for interactive diagnostic reasoning. The study systematically reveals, for the first time, that static evaluations substantially overestimate LLMs’ true diagnostic proficiency and proposes a complementary interactive evaluation paradigm. Integrating multi-turn dialogue interaction with dual-dimensional assessment of diagnostic accuracy and evidential quality, experiments across 468 clinical cases and 15 models show that multi-turn evidence collection leads to a 12.75% drop in diagnostic accuracy and a 24.36% reduction in supporting evidence quality.

This study addresses the frequent violations of clinical coding standards—such as ICD-10, CPT, and HL7 FHIR—by large language models when generating structured medical data, which impedes integration with electronic health record systems. To mitigate this, the authors propose and validate a closed-loop verification-and-repair framework that automatically detects and iteratively corrects formatting errors. The approach is evaluated using three open-source models—Qwen2.5-7B, Llama3.1-8B, and Gemma2-9B—deployed locally across 320 clinical scenarios. Results demonstrate a substantial improvement in schema compliance across all models, achieving an overall adherence rate of 99.0% and increasing individual model performance by 7.8 to 12.5 percentage points. Notably, 96% of detected errors were attributable to repairable representation-layer issues, with most resolved within one or two correction rounds, effectively compensating for the models’ limited understanding of healthcare IT standards.

This study addresses the challenge of error-prone manual verification of tables, figures, and listings (TFLs) in clinical trial reports, which often fails to detect structural or logical inconsistencies. The authors propose PROVE, a novel framework that leverages large language models (LLMs) for semantic parsing and evidence tracing of TFL content, integrated with a programmable rule engine to perform deterministic numerical and logical validation against SDTM/ADaM standards. Designed as a multi-agent architecture, PROVE combines LLM-driven semantic understanding with rule-based checks to enable auditable, configurable automated cross-verification. The approach achieves 100% accuracy under exact label matching; when confronted with linguistic variations, LLM assistance boosts recall from 0.588 to 0.993 and F1 score from 0.735 to 0.996.

Current static, single-turn evaluations inadequately capture the comprehensive capabilities of large language models in dynamic clinical settings—particularly their ability to gather information, formulate treatment plans, and manage long-term care. To address this gap, this work proposes MedSP1000, the first interactive evaluation benchmark built upon standardized patient teaching cases. It leverages a closed-loop simulation system comprising structured scenario scripts, an environment controller, and patient agents to enable end-to-end, automatically scored assessments, augmented by expert-reviewed, trajectory-level scoring criteria. Experiments reveal that even the strongest general-purpose model (GPT-5.5) achieves only 60.4% of expert-rated performance, while domain-specific medical models perform substantially worse (40.0%). Moreover, increased reasoning compute yields no significant gains, exposing critical clinical failure modes invisible to static benchmarks and underscoring that current models remain unsafe for real-world clinical deployment.

While current large language models demonstrate accuracy in clinical diagnosis, it remains unclear whether their reasoning follows stable, structured clinical logic. This work proposes the Clinical Reasoning Graph framework—a structured graph representation grounded in a clinical ontology comprising five node types and seven edge types—and leverages natural language processing and graph similarity metrics to extract and analyze 750 diagnostic trajectories. The study reveals that graph similarity between correct and incorrect diagnoses is nearly identical (0.488 vs. 0.484), and reasoning structures show no significant consistency across similar cases, indicating a lack of schematic-level stability in cross-case reasoning. Although structured reflection prompts improve feature analysis, they do not enhance structural consistency. These findings underscore the need for process-level evaluation to complement conventional outcome-based accuracy and offer a novel paradigm for explainability in clinical AI.