Evaluating Multi-Turn Multimodal Diagnostic Reasoning on Challenging Real-World Clinical Cases

2026-07-28

AI Summary

Existing clinical diagnostic evaluation methods struggle to capture the true complexity of progressive multimodal information disclosure and dynamic reasoning. To address this gap, this work proposes ClinMM-Bench—the first large-scale, multi-turn, multimodal diagnostic benchmark grounded in real-world clinical scenarios—encompassing 1,089 challenging cases and 3,760 medical images, along with a dual-layer evaluation framework that systematically assesses both diagnostic accuracy and reasoning quality. Evaluation of 15 state-of-the-art multimodal large language models reveals that, although closed-source models achieve the best performance, their fully correct diagnosis rate remains limited; while models can generate plausible hypotheses, they exhibit significant deficiencies in constructing reliable and coherent reasoning chains. The study further identifies five representative failure patterns, offering concrete directions for future model improvement.

Abstract

Clinical diagnostic evaluation should not only assess whether models can provide correct diagnoses, but also reflect the realities of clinical practice, including progressive disclosure of multimodal information, dynamic updating of diagnostic hypotheses, and continuous refinement of clinical reasoning. However, existing evaluations of multimodal large language models (MLLMs) typically rely on single-turn or isolated tasks, making it difficult to fully capture the complexity of real-world clinical diagnosis. To bridge this gap, we developed ClinMM-Bench, the largest multi-turn multimodal clinical diagnostic evaluation benchmark to date. ClinMM-Bench contains 1,089 challenging real-world clinical cases and 3,760 medical images across eight specialties. We systematically evaluated 15 representative MLLMs using a two-level evaluation framework that assessed both diagnostic accuracy and diagnostic reasoning quality. Results showed that proprietary models achieved the highest overall diagnostic accuracy, but the proportion of completely correct diagnoses remained limited across all models. In terms of diagnostic reasoning quality, current models can identify plausible diagnostic directions but still have considerable limitations in generating reliable diagnostic reasoning. Error analysis further identified five representative failure modes: information synthesis failure, knowledge mapping error, perception error, premature closure, and visual hallucination.

Problem

• multimodal large language models
• clinical diagnosis
• multi-turn evaluation
• diagnostic reasoning
• real-world clinical cases

Innovation

• multi-turn evaluation
• multimodal diagnostic reasoning
• ClinMM-Bench
• clinical reasoning quality
• failure mode analysis