SwinTextUNet: Integrating CLIP-Based Text Guidance into Swin Transformer U-Nets for Medical Image Segmentation

2026-04-10

AI Summary

This study addresses the limited performance of conventional medical image segmentation models in ambiguous or low-contrast regions by proposing a novel multimodal feature alignment strategy that integrates CLIP-generated textual semantic embeddings into a Swin Transformer U-Net architecture. The approach uniquely combines a cross-attention mechanism with a convolutional fusion module to effectively align high-level semantic guidance from text with multi-scale visual features, thereby enhancing the model’s anatomical understanding and segmentation robustness. Evaluated on the QaTa-COV19 dataset, the proposed four-stage variant achieves a Dice score of 86.47% and an IoU of 78.2%, demonstrating its superior efficacy and validity.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Multimodal Learning
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Precise medical image segmentation is fundamental for enabling computer aided diagnosis and effective treatment planning. Traditional models that rely solely on visual features often struggle when confronted with ambiguous or low contrast patterns. To overcome these limitations, we introduce SwinTextUNet, a multimodal segmentation framework that incorporates Contrastive Language Image Pretraining (CLIP), derived textual embeddings into a Swin Transformer UNet backbone. By integrating cross attention and convolutional fusion, the model effectively aligns semantic text guidance with hierarchical visual representations, enhancing robustness and accuracy. We evaluate our approach on the QaTaCOV19 dataset, where the proposed four stage variant achieves an optimal balance between performance and complexity, yielding Dice and IoU scores of 86.47% and 78.2%, respectively. Ablation studies further validate the importance of text guidance and multimodal fusion. These findings underscore the promise of vision language integration in advancing medical image segmentation and supporting clinically meaningful diagnostic tools.

Problem

• medical image segmentation
• low contrast
• ambiguous patterns
• visual features
• precision

Innovation

• CLIP
• Swin Transformer
• medical image segmentation
• multimodal fusion
• text guidance