A Smart Multimodal Healthcare Copilot with Powerful LLM Reasoning

2025-06-03

AI Summary

Frequent misdiagnoses escalate healthcare costs and compromise patient safety. To address this, we propose MedRAG—the first intelligent multimodal clinical decision support system integrating voice-based patient interviews, unstructured clinical notes, and structured electronic health records (EHRs) to generate diagnostic hypotheses, treatment plans, medication recommendations, and follow-up questions. Its core innovation is a knowledge graph–guided retrieval-augmented generation (KG-RAG) framework that enables medical-logic-driven, cross-modal reasoning over heterogeneous data sources, markedly improving diagnostic interpretability and clinical applicability. Evaluated on multiple public and proprietary medical datasets, MedRAG consistently outperforms state-of-the-art baselines in recommendation accuracy and clinical specificity. The source code and demonstration video are publicly available.

Technology Category

• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Misdiagnosis causes significant harm to healthcare systems worldwide, leading to increased costs and patient risks. MedRAG is a smart multimodal healthcare copilot equipped with powerful large language model (LLM) reasoning, designed to enhance medical decision-making. It supports multiple input modalities, including non-intrusive voice monitoring, general medical queries, and electronic health records. MedRAG provides recommendations on diagnosis, treatment, medication, and follow-up questioning. Leveraging retrieval-augmented generation enhanced by knowledge graph-elicited reasoning, MedRAG retrieves and integrates critical diagnostic insights, reducing the risk of misdiagnosis. It has been evaluated on both public and private datasets, outperforming existing models and offering more specific and accurate healthcare assistance. A demonstration video of MedRAG is available at: https://www.youtube.com/watch?v=PNIBDMYRfDM. The source code is available at: https://github.com/SNOWTEAM2023/MedRAG.

Problem

• Reducing misdiagnosis risks in healthcare systems
• Enhancing medical decision-making with multimodal inputs
• Improving diagnostic accuracy using LLM and knowledge graphs

Innovation

• Multimodal input with voice and EHR
• Retrieval-augmented generation with knowledge graphs
• LLM reasoning for medical decision-making