Aligning Multimodal Patient Evidence with Biomedical Knowledge Graphs for Clinical LLMs

2026-10-05

AI Summary

This study addresses the lack of explicit linkage between multimodal evidence and biomedical knowledge in clinical prediction, which renders reasoning untraceable. To this end, we propose the MM-KG framework, which constructs a hierarchical graph and employs a routing-priority aligner to explicitly align heterogeneous patient data with the UMLS knowledge graph. Furthermore, it integrates query-driven subgraph retrieval to augment large language model reasoning. The core innovation lies in establishing an evidence-linking mechanism that is traceable, testable, and amenable to modular ablation. Experimental results demonstrate that the proposed method significantly improves AUROC on the MIMIC and ADNI datasets, outperforms the MindMap baseline in ranking performance, and achieves higher accuracy with less context.

Abstract

Clinical questions often depend on linking a patient's multimodal evidence to external biomedical knowledge, yet existing predictive systems rarely represent such links explicitly, so they can neither be traced to their evidence sources nor removed to measure their contributions. We present MM-KG (Multimodal Knowledge Graph), which represents heterogeneous, multimodal patient observations and biomedical concepts as separate layers in one typed graph, joined by explicit alignment edges. First, modality-specific harmonizers convert EHR text, imaging, genomic, and biospecimen data into typed observations mapped to UMLS concepts, which a route-prioritized aligner links to a biomedical knowledge graph. Query-conditioned retrieval then selects a compact subgraph for downstream use by a large language model or a graph neural network. We build MM-KGs for MIMIC-IV and ADNI, and evaluate them with a 2x2 design that separates patient evidence, biomedical knowledge, and their interaction. On questions that require both sources, neither source alone performs far above chance, whereas their combination yields a drug-controlled AUROC interaction of +0.194 on MIMIC and +0.299 on ADNI. On held-out five-candidate ranking, MM-KG outperforms MindMap by +0.131 Hits@1 and leads an adapted GraphCare on the items that require consulting the patient, and deleting the single answer-bearing relation from the retrieved packet returns Hits@1 to the no-knowledge baseline. Finally, query-conditioned retrieval reaches 0.731 AUROC with 6.8x less context than the strongest generic policy, whereas static knowledge graph context gives no consistent gain on ordinary outcome prediction. Knowledge graphs thus benefit clinical LLMs not as background context but as explicit links between multimodal patient evidence and the relation a question requires, and MM-KG makes these links retrievable, traceable, and testable.

Problem

• Multimodal patient evidence
• Biomedical knowledge graphs
• Clinical LLMs
• Knowledge alignment
• Traceability

Innovation

• Multimodal Knowledge Graph
• Explicit Alignment
• Query-conditioned Retrieval
• Clinical LLMs
• Traceability