MedDDC-Eval: Diagnosis-Decoupled Evaluation of Multi-Turn Medical Consultation Agents

2026-07-21

AI Summary

This work addresses the challenge in evaluating medical diagnostic agents, where inquiry strategies and diagnosis generation are typically entangled, preventing independent assessment of inquiry quality. To resolve this, the authors propose a diagnosis-decoupled evaluation framework that isolates inquiry history as the primary object of comparison by fixing the diagnostic mapping module. The framework introduces an auditable metric system encompassing diagnostic accuracy, trajectory coherence, and inquiry efficiency, supported by a directional semantic coverage matching mechanism to enable controllable attribution and interpretable measurement. Key technical components include a frozen diagnostic reader, deterministic one-to-one matching, and Group Relative Policy Optimization (GRPO). Experiments show that the optimized Qwen3-32B inquiry strategy achieves score improvements of 9.7 and 4.6 on the Record and Dialogue test sets, respectively; moreover, merely swapping the diagnostic reader alters F1 scores by 2.2–19.0 points and reverses 18%–36% of strategy rankings.

Abstract

Multi-turn medical consultation agents must decide what to ask, adapt to patient responses, and determine when the collected evidence is sufficient. However, coupled evaluation conflates the quality of the policy-elicited history with policy-specific terminal diagnosis generation: strong generation can compensate for a thin history, while weaker generation can obscure a rich one. We introduce MedDDC-Eval, a diagnosis-decoupled testbed that treats elicited history as the comparison object and holds the history-to-diagnosis mapping constant through a shared frozen reader. Across two held-out sources, a grounded interface and an auditable diagnosis-trajectory-efficiency (D/T/E) harness measure diagnostic usefulness, information acquisition, and efficiency. Directional semantic coverage followed by deterministic one-to-one assignment yields coherent precision-recall counts for open-ended items, with at most one credited match per prediction or reference. Holding histories fixed, changing only the diagnostic reader shifts diagnosis F1 by 2.2-19.0 points and reverses 18% and 36% of pairwise policy orderings on the Record and Dialogue splits. We further apply standard Group Relative Policy Optimization (GRPO) over interactive multi-turn rollouts to post-train Qwen3-32B using diagnosis-result and trajectory feedback. On the 100-case Record and 70-case Dialogue splits, the trained policy improves over its initialization by 9.7 and 4.6 total-score points; removing either primary signal lowers held-out joint performance. These results show that MedDDC-Eval supports controlled attribution, interpretable elicited-history measurement, and evaluation-guided evidence-acquisition policy development.

Problem

• medical consultation agents
• multi-turn dialogue
• diagnosis-decoupled evaluation
• elicited history
• policy evaluation

Innovation

• diagnosis-decoupled evaluation
• frozen reader
• diagnostic trajectory-efficiency
• directional semantic coverage
• GRPO