CARE-Bench: Benchmarking Patient-Facing LLM Triage

2026-08-04

AI Summary

This work addresses the safety and accuracy of next-step action recommendations by patient-facing medical large language models during symptom consultation, introducing the CARE-Bench benchmark. CARE-Bench formulates patient triage as a sequential, action-oriented task comprising four categories of actions, leveraging 500 clinical cases and 1,059 patient statement prefixes. It evaluates 11 models under zero-shot and minimal-prompt settings using source-grounded data reconstruction, GPT-5.5 response mapping, and macro F1 scores. Results show that prompting improves macro F1 from 31.2–50.4 to 46.9–63.4; however, only 33.5% of models correctly request clarification when needed, revealing a critical deficiency in current models’ ability to judge appropriate timing for information-seeking actions.

Abstract

Patient-facing medical LLMs and agents increasingly answer symptom questions before clinician contact, where the key safety question is what action the user should take next. We introduce CARE-Bench, a source-grounded benchmark that evaluates sequential patient-facing triage as a four-label per-turn current-action task. CARE-Bench contains 500 cases and 1,059 evaluated patient-disclosure prefixes reconstructed from medical dialogue, consultation, and follow-up-question sources. We evaluate 11 models on 269 held-out rounds under unprompted and minimally prompted open-ended protocols, using a fixed GPT-5.5 mapper to code each response into the four-label action space. Unprompted macro-F1 remains low, ranging from 31.2 to 50.4. Prompting improves 10 of 11 models, with prompted macro-F1 ranging from 46.9 to 63.4, but substantial threshold errors remain. Prompted models often recommend care before needed clarification is obtained; when the correct action was to ask for more information, only 33.5% of prompted outputs preserved the step. The persistence of these errors after prompting suggests that patient-facing triage is not a simple prompting problem and supports explicit evaluation of action timing before deployment.

Problem

• patient-facing LLM
• triage
• action timing
• safety evaluation
• medical benchmark

Innovation

• CARE-Bench
• patient-facing LLM triage
• action timing evaluation
• source-grounded benchmark
• sequential medical dialogue