RESPClinBench: Benchmarking Multimodal Clinical Decision-Making and Longitudinal Disease Management in Respiratory Specialty Care

2026-08-05

AI Summary

Current evaluations of medical large language models predominantly focus on exam-style question answering, which inadequately captures the multimodal interpretation, longitudinal risk assessment, and comprehensive care management required in respiratory medicine. To address this gap, this work proposes RESPClinBench—the first clinical scenario-based benchmark specifically designed for respiratory specialties—featuring tasks on acute exacerbation of chronic obstructive pulmonary disease (AECOPD-PIM) and pulmonary nodules (PNBIM), integrating multimodal imaging, structured clinical data, and longitudinal follow-up information. The benchmark introduces a novel tripartite evaluation framework combining atomic action recall, LLM-as-a-Judge scoring, and independent safety risk tagging, with systematic assessment conducted via standardized APIs, deterministic generation, and expert review. Evaluated on 623 real-world cases, Qwen3.6-27B achieved the highest performance (mean score: 68.58), yet exhibited significant image hallucination (31.85%) and medication-related safety concerns (26.93%).

Abstract

Background: Respiratory specialty care requires multimodal interpretation, longitudinal risk assessment, guideline-concordant intervention, and whole-course management, which are poorly represented by examination-oriented medical benchmarks. Objective: To develop RESPClinBench, a real-world scenario-based benchmark for respiratory clinical decision-making, and evaluate seven contemporary large language models across AECOPD-PIM and PNBIM. Methods: RESPClinBench cases were adapted from de-identified respiratory clinical data. Three attending-level respiratory physicians revised cases, reference answers, and atomic clinical-action points, while one senior respiratory specialist performed cross-review and final adjudication. AECOPD-PIM comprised 427 open-ended COPD cases, and PNBIM comprised 196 multimodal pulmonary nodule cases combining chest CT with structured clinical information. Seven models generated 4,361 responses through standardized API inference with temperature 0 and a maximum output length of 8192 tokens. An automated framework calculated the final score as the arithmetic mean of atomic-action recall and rubric-based LLM-as-a-Judge assessment. Results: Across 623 cases, the mean final score was 68.58. Qwen3.6-27B ranked first overall at 71.22, Qwen3.5-397B-A17B led PNBIM at 72.48, and Qwen3.6-27B led AECOPD-PIM at 71.11. Imaging hallucination and serious medical risk occurred in 31.85% and 8.16% of PNBIM responses; medication-safety risk and serious medical risk occurred in 26.93% and 1.44% of AECOPD-PIM responses. Conclusions: RESPClinBench identifies task-specific limitations in multimodal pulmonary nodule assessment and longitudinal COPD management. Combining explicit clinical-action coverage, holistic evaluation, and independent safety flags provides a clinically grounded basis for model selection and prospective validation.

Problem

• multimodal clinical decision-making
• longitudinal disease management
• respiratory specialty care
• clinical benchmarking
• real-world scenario

Innovation

• multimodal clinical benchmark
• longitudinal disease management
• clinical-action recall
• LLM-as-a-Judge
• medical safety evaluation