ResidencyRL: Reinforcement Learning in Simulated Clinical Environments

2026-08-07

AI Summary

This work addresses the limitations of existing clinical AI systems, which struggle to optimize full decision sequences across multi-turn physician–patient dialogues and lack the capacity for progressive clinical reasoning akin to that of resident physicians. The authors propose a closed-loop training paradigm that integrates multi-turn reinforcement learning with an adversarial large language model (LLM)-based patient simulator, enabling AI agents to learn complex diagnostic workflows involving up to 60 dialogue rounds and multiple tool invocations in a high-fidelity environment. A structured, multi-dimensional reward mechanism—encompassing diagnostic accuracy, safety, and other clinical criteria—significantly enhances overall performance: the approach achieves a 7.0% improvement in diagnostic accuracy, a 31% reduction in missed critical alerts, and an 87.6% preference rate in expert blind evaluations, while consistently outperforming state-of-the-art methods on unseen benchmarks such as AMIE and AgentClinic.

Abstract

In medical education, physicians convert academic knowledge into clinical expertise through residency: years of training across thousands of encounters, with diverse sources of feedback and progressively greater autonomy. Much of clinical reasoning relies on the patient encounter, a dialogue in which a clinician elicits history, refines diagnostic hypotheses, and decides management under uncertainty. While large language models (LLMs) excel on static medical benchmarks, methods to optimize the full sequence of clinical decisions remain underdeveloped. We present ResidencyRL, a reinforcement learning (RL) method for training clinical artificial intelligence (AI) agents through simulated multi-turn clinical encounters (up to 60 dialogue turns and 8 tool calls per trajectory). ResidencyRL pairs the policy agent with LLM simulators capable of complex, adversarial behaviors, training against a structured reward aligned to diagnostic accuracy, management quality, communication, documentation, and safety. On held-out evaluations, the ResidencyRL agent improves diagnostic accuracy by 7.0% under adversarial conditions (88.0% vs. 81.0%) and reduces missed red flag rates by 31%, demonstrating rigorous mitigation of premature closure. Blinded expert clinicians validated these gains, preferring the trained agent in 87.6% of side-by-side comparisons. The procedural competencies transfer to unseen benchmarks: the agent outperforms the base model across all six clinical axes of the AMIE multi-visit benchmark, and shows consistent directional improvements on AgentClinic and CRAFT-MD. Our findings demonstrate that sequential clinical decision-making can be effectively learned through multi-turn RL in simulation, yielding robust, generalizable capabilities, paving the way towards clinical mastery. Prospective validation with real-world workflows remains necessary to establish clinical utility.

Problem

• clinical decision-making
• reinforcement learning
• simulated clinical environments
• sequential reasoning
• medical AI

Innovation

• Reinforcement Learning
• Clinical Simulation
• Large Language Models
• Diagnostic Reasoning
• Adversarial Training