Localization-Infused Vision-Language Semantic Fusion for Text-Guided Medical Image Segmentation

2026-07-15

AI Summary

Existing text-guided medical image segmentation methods struggle to explicitly model target location information from textual descriptions and lack effective mechanisms for multi-level semantic fusion. To address these limitations, this work proposes the LoG framework, which, for the first time, explicitly incorporates spatial localization cues into the vision–language fusion process. Specifically, a multi-scale localization prediction module extracts spatial semantics from text, and a three-tier localization-enhanced fusion mechanism is introduced—comprising localization-guided feature fusion, localization-gated attention, and a localization-constrained loss. The proposed method achieves Dice scores of 91.59%, 80.71%, and 94.59% on QaTa-COV19, MosMedData+, and Kvasir-SEG, respectively, significantly outperforming current state-of-the-art approaches and demonstrating the efficacy of localization guidance in enhancing text-driven semantic understanding for medical image segmentation.

Abstract

Medical image segmentation is essential for modern computer-aided medicine. Recently, text-guided segmentation has shown promise by incorporating clinician-formulated textual reports as semantic guidance for image segmentation. These reports describe target appearance, location, and neighboring anatomy, providing explicit guidance for localization and delineation. Existing text-guided segmentation methods typically extract textual semantics implicitly through a pretrained text encoder and then integrate vision-language semantics via straightforward image-text feature fusion. However, these methods do not explicitly capture target-oriented information embedded in textual reports, particularly target location, and do not explore multi-level information fusion strategies beyond basic feature-level fusion, limiting the extraction and integration of critical textual semantics. In this study, we propose LoG, a localization-infused vision-language fusion framework for text-guided medical image segmentation. By jointly performing multi-scale target localization tasks, LoG explicitly captures target-oriented vision-language semantics and enables three-level localization-infused semantic fusion: (i) localization-guided feature fusion that directly infuses location-relevant semantics into visual features, (ii) localization-gated attention fusion that redirects multi-scale localization predictions to reinforce critical regions, and (iii) localization-constrained loss fusion that supervises segmentation based on spatial consistency with target localization. Extensive experiments on three benchmark datasets, involving three medical imaging modalities with paired textual reports, demonstrate that LoG achieves Dice scores of 91.59%, 80.71%, and 94.59% on QaTa-COV19, MosMedData+, and Kvasir-SEG, respectively, consistently outperforming state-of-the-art medical image segmentation methods.

Problem

• text-guided segmentation
• medical image segmentation
• vision-language fusion
• target localization
• semantic fusion

Innovation

• localization-infused fusion
• text-guided segmentation
• vision-language semantic fusion
• multi-scale localization
• medical image segmentation