KG-SAM: Injecting Anatomical Knowledge into Segment Anything Models via Conditional Random Fields

2025-09-25

AI Summary

SAM faces critical challenges in medical image segmentation, including ill-defined boundaries, insufficient modeling of anatomical relationships, and lack of uncertainty quantification. To address these, we propose KG-SAM—a novel framework that first integrates an anatomical knowledge graph (KG) and an energy-driven conditional random field (CRF) into the SAM architecture. The KG encodes prior anatomical structural constraints, while the CRF jointly optimizes segmentation boundaries and models prediction uncertainty. An uncertainty-aware fusion module further enhances robustness. This design significantly improves segmentation consistency, interpretability, and clinical applicability. Extensive evaluation on multi-center datasets demonstrates state-of-the-art performance: Dice scores of 82.69% for prostate segmentation, and 78.05% and 79.68% for abdominal MRI and CT segmentation, respectively—each substantially surpassing existing methods.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Calibration & Uncertainty Quantification
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

While the Segment Anything Model (SAM) has achieved remarkable success in image segmentation, its direct application to medical imaging remains hindered by fundamental challenges, including ambiguous boundaries, insufficient modeling of anatomical relationships, and the absence of uncertainty quantification. To address these limitations, we introduce KG-SAM, a knowledge-guided framework that synergistically integrates anatomical priors with boundary refinement and uncertainty estimation. Specifically, KG-SAM incorporates (i) a medical knowledge graph to encode fine-grained anatomical relationships, (ii) an energy-based Conditional Random Field (CRF) to enforce anatomically consistent predictions, and (iii) an uncertainty-aware fusion module to enhance reliability in high-stakes clinical scenarios. Extensive experiments across multi-center medical datasets demonstrate the effectiveness of our approach: KG-SAM achieves an average Dice score of 82.69% on prostate segmentation and delivers substantial gains in abdominal segmentation, reaching 78.05% on MRI and 79.68% on CT. These results establish KG-SAM as a robust and generalizable framework for advancing medical image segmentation.

Problem

• Addresses ambiguous boundaries in medical image segmentation
• Incorporates anatomical knowledge to improve segmentation consistency
• Provides uncertainty estimation for clinical reliability

Innovation

• Integrates medical knowledge graph for anatomical relationships
• Uses energy-based CRF for anatomically consistent predictions
• Implements uncertainty-aware fusion module for reliability