Evaluating Large Language Models in Dynamic Clinical Decision-Making with Standardized Patient Cases

2026-06-03

AI Summary

Current static, single-turn evaluations inadequately capture the comprehensive capabilities of large language models in dynamic clinical settings—particularly their ability to gather information, formulate treatment plans, and manage long-term care. To address this gap, this work proposes MedSP1000, the first interactive evaluation benchmark built upon standardized patient teaching cases. It leverages a closed-loop simulation system comprising structured scenario scripts, an environment controller, and patient agents to enable end-to-end, automatically scored assessments, augmented by expert-reviewed, trajectory-level scoring criteria. Experiments reveal that even the strongest general-purpose model (GPT-5.5) achieves only 60.4% of expert-rated performance, while domain-specific medical models perform substantially worse (40.0%). Moreover, increased reasoning compute yields no significant gains, exposing critical clinical failure modes invisible to static benchmarks and underscoring that current models remain unsafe for real-world clinical deployment.

Abstract

Large language models (LLMs) are increasingly proposed as clinical agents, yet static, single-turn benchmarks cannot capture how a model dynamically delivers care across an encounter: gathering information, planning treatment, and adapting longitudinal management across successive patient states. Medical education has long addressed an analogous challenge through standardized patients (SPs): trained actors who consistently portray clinical cases, enabling realistic practice and objective, scripted assessment. Here we introduce MedSP1000, an SP-derived interactive benchmark for clinical-agent evaluation, including 1,638 SP cases with 24,602 trajectory-level peer-reviewed rubrics. MedSP1000 converts peer-reviewed SP teaching cases into executable scenarios with defined SP case scripts, clinical environment contexts, and human-validated structured rubric. In each simulation evaluation run, a clinical agent interacts in closed loop with a patient agent and an environment controller, and its behaviour is scored throughout the encounter against expert criteria specified in the original materials. Applying MedSP1000 to a range of general-purpose and medically specialized LLMs, we find that performance on static benchmarks does not reliably translate to such educational scenarios. The best-performing model, GPT-5.5, completes only 60.4% of expert-defined rubric items, whereas the strongest medically specialized model reaches 40.0%; increasing test-time compute produces no measurable gain. These results suggest that current LLMs, including agentic systems tuned for medicine, are not yet reliable enough to be safely integrated into actual clinical practice. More broadly, MedSP1000 shows how process-level, SP-style evaluation can reveal clinically relevant failure modes that single-turn benchmarks miss.

Problem

• dynamic clinical decision-making
• large language models
• standardized patients
• interactive evaluation
• clinical benchmarking

Innovation

• standardized patients
• dynamic clinical evaluation
• interactive benchmark
• clinical decision-making
• large language models