MedClarify: An information-seeking AI agent for medical diagnosis with case-specific follow-up questions

2026-02-19

AI Summary

This work addresses the challenge that existing medical large language models struggle with differential diagnosis in cases of incomplete clinical information, often leading to high error rates. The authors propose an information-theoretic AI agent that actively formulates targeted follow-up questions to mimic the clinical diagnostic reasoning process, dynamically reducing diagnostic uncertainty by maximizing information gain. This approach introduces, for the first time, an information-gain-guided active questioning mechanism into medical diagnosis: it leverages a large language model to generate a candidate differential diagnosis set and selects the optimal follow-up question to maximize uncertainty reduction. Experimental results demonstrate that, compared to single-turn question-answering baselines, the proposed method reduces diagnostic error rates by 27 percentage points, substantially improving diagnostic accuracy in scenarios with incomplete patient information.

Technology Category

• Machine Learning: Active Learning
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning
• Natural Language Processing: Question Answering

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Large language models (LLMs) are increasingly used for diagnostic tasks in medicine. In clinical practice, the correct diagnosis can rarely be immediately inferred from the initial patient presentation alone. Rather, reaching a diagnosis often involves systematic history taking, during which clinicians reason over multiple potential conditions through iterative questioning to resolve uncertainty. This process requires considering differential diagnoses and actively excluding emergencies that demand immediate intervention. Yet, the ability of medical LLMs to generate informative follow-up questions and thus reason over differential diagnoses remains underexplored. Here, we introduce MedClarify, an AI agent for information-seeking that can generate follow-up questions for iterative reasoning to support diagnostic decision-making. Specifically, MedClarify computes a list of candidate diagnoses analogous to a differential diagnosis, and then proactively generates follow-up questions aimed at reducing diagnostic uncertainty. By selecting the question with the highest expected information gain, MedClarify enables targeted, uncertainty-aware reasoning to improve diagnostic performance. In our experiments, we first demonstrate the limitations of current LLMs in medical reasoning, which often yield multiple, similarly likely diagnoses, especially when patient cases are incomplete or relevant information for diagnosis is missing. We then show that our information-theoretic reasoning approach can generate effective follow-up questioning and thereby reduces diagnostic errors by ~27 percentage points (p.p.) compared to a standard single-shot LLM baseline. Altogether, MedClarify offers a path to improve medical LLMs through agentic information-seeking and to thus promote effective dialogues with medical LLMs that reflect the iterative and uncertain nature of real-world clinical reasoning.

Problem

• medical diagnosis
• follow-up questions
• differential diagnosis
• diagnostic uncertainty
• large language models

Innovation

• information-seeking agent
• differential diagnosis
• follow-up questioning
• information gain
• medical LLM