ClinicalMC: A Benchmark for Multi-Course Clinical Decision-Making with Large Language Models

2026-06-02

AI Summary

Existing large language models lack systematic evaluation in multi-turn clinical decision-making scenarios that span the entire patient journey from admission to discharge. To address this gap, this work proposes ClinicalMC—the first bilingual (Chinese-English), multi-turn clinical benchmark encompassing 7,079 high-quality cases with an average of 3.42–5.11 dialogue turns per case. By integrating a multi-agent simulation framework—emulating patients, physicians, and examiners—and structured clinical pathway modeling, ClinicalMC enables unified, reproducible evaluation of both closed-source and open-source models, including domain-specific medical LLMs, under both single-turn static and multi-turn dynamic settings. The benchmark establishes a cross-lingual, cross-architectural quantitative evaluation framework for assessing multi-turn clinical reasoning capabilities, as demonstrated through evaluations of models such as GPT-5-mini, DeepSeek-V3.2, and HuatuoGPT-o1.

Abstract

Large language models (LLMs) have been widely adopted in healthcare, yet they still encounter significant challenges in complex clinical decision-making scenarios. Existing benchmarks primarily assess LLM performance in single-course settings and lack systematic evaluation in multi-course scenarios, where a patient's condition evolves over time. To address this gap, we propose ClinicalMC, a benchmark for multi-course clinical decision-making. It includes 1,275 Chinese and 5,804 English samples across four stages from admission to discharge. These stages cover triage, first-course examination/diagnosis/treatment, subsequent multi-course examination/assessment/treatment, and final diagnosis. In ClinicalMC, patients in the English dataset undergo an average of 5.11 clinical courses, whereas those in the Chinese dataset undergo 3.42. To assess LLM performance, we construct a multi-agent evaluation framework that includes patient, examiner, and doctor agents. Based on the benchmark and framework, we design two experimental settings -- a single-turn static setting and a multi-turn dynamic setting -- and assess three categories of LLMs: 1) closed-source LLMs like GPT5-mini; 2) open-source LLMs like DeepSeek-V3.2; and 3) medical LLMs like HuatuoGPT-o1. Through extensive evaluation, we aim to better understand LLM performance in the medical domain and support its effective deployment in healthcare.

Problem

• multi-course clinical decision-making
• large language models
• clinical benchmark
• healthcare AI
• longitudinal patient care

Innovation

• multi-course clinical decision-making
• LLM benchmark
• multi-agent evaluation framework
• dynamic clinical simulation
• medical large language models