MultiTurnPSB: Evaluating Multi-Turn Jailbreak Attacks an dClassifier-Based Defenses for Medical AI Safety

2026-05-30

AI Summary

This study addresses a critical gap in medical AI safety evaluation, which typically relies on single-turn interactions and fails to capture security degradation induced by multi-turn jailbreak attacks—such as user persistence, appeals to urgency, or invocation of authority after initial refusals. The authors propose MultiTurnPSB, a four-turn adversarial benchmark that systematically evaluates the safety of large medical language models under fixed templates, adaptive strategies, and real-time human red-teaming. They also introduce a lightweight input classifier for defense. Their analysis reveals, for the first time, safety degradation trajectories missed by single-turn assessments, identifies a two-factor attack pattern leading to catastrophic failures, and demonstrates that safety training can generalize across attacker personas. Experiments show GPT-4.1-mini’s unsafe response rate surges from 35% to nearly 80% over four turns; the classifier reduces this by 52 percentage points but incurs a 45% false positive rate, whereas Claude Sonnet increasingly refuses to answer in later turns, exhibiting role-generalized robustness.

Abstract

Patient-facing medical chatbots are commonly evaluated on single-turn prompts, yet real users push back after refusals, add urgency, and invoke authority. We introduce MultiTurnPSB, a four-turn adversarial extension of PatientSafetyBench, and evaluate GPT-4.1-mini under fixed template, template-adaptive, and live adversarial attacks. Unsafe responses rise from 35% to nearly 80% by Turn 4 under live attack. Under the same adversary, GPT-4.1-mini and Claude Sonnet 4.5 are statistically indistinguishable at baseline but diverge to a 19x gap by Turn 4, a difference invisible to single-turn evaluation. We characterize four degradation trajectory signatures and identify a two-element attack formula responsible for most catastrophic failures. A lightweight input-side classifier reduces Turn 4 unsafe responses by 52 percentage points despite severe accuracy degradation, but the 45% false alarm rate on benign queries is the primary deployment constraint. A methodological finding also emerges: Claude Sonnet refused to generate adversarial messages in over half of late-turn conversations despite explicit red team framing, suggesting safety training may generalize to the attacker role.

Problem

• multi-turn jailbreak
• medical AI safety
• adversarial attacks
• chatbot safety
• classifier-based defenses

Innovation

• multi-turn jailbreak
• adversarial evaluation
• medical AI safety
• input-side classifier
• safety generalization