Segment Anything in Pathology Images with Natural Language

2025-06-26

AI Summary

Pathological image segmentation is hindered by scarce expert annotations and rigid class definitions, limiting clinical deployment. To address this, we propose PathSegmentor—the first foundational model for text-prompted segmentation in pathology—introducing natural language instructions to enable fine-grained semantic segmentation without spatial prompts (e.g., points or bounding boxes). We construct PathSeg, the largest multi-source pathological segmentation dataset to date, comprising 275k image-mask-label triplets spanning 160 tissue structures. PathSegmentor employs a lightweight Transformer architecture integrating cross-modal alignment, large-scale self-supervised pretraining, and multi-center fine-tuning. On Dice score, it outperforms state-of-the-art spatial-prompt and text-prompt methods by 0.145 and 0.429, respectively, significantly improving segmentation accuracy for complex structures, cross-institutional generalizability, and result interpretability—thereby advancing biomarker discovery and clinical decision support.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Semantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semantics
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Pathology image segmentation is crucial in computational pathology for analyzing histological features relevant to cancer diagnosis and prognosis. However, current methods face major challenges in clinical applications due to limited annotated data and restricted category definitions. To address these limitations, we propose PathSegmentor, the first text-prompted segmentation foundation model designed specifically for pathology images. We also introduce PathSeg , the largest and most comprehensive dataset for pathology segmentation, built from 17 public sources and containing 275k image-mask-label triples across 160 diverse categories. With PathSegmentor, users can perform semantic segmentation using natural language prompts, eliminating the need for laborious spatial inputs such as points or boxes. Extensive experiments demonstrate that PathSegmentor outperforms specialized models with higher accuracy and broader applicability, while maintaining a compact architecture. It significantly surpasses existing spatial- and text-prompted models by 0.145 and 0.429 in overall Dice scores, respectively, showing strong robustness in segmenting complex structures and generalizing to external datasets. Moreover, PathSegmentor's outputs enhance the interpretability of diagnostic models through feature importance estimation and imaging biomarker discovery, offering pathologists evidence-based support for clinical decision-making. This work advances the development of explainable AI in precision oncology.

Problem

• Limited annotated data in pathology image segmentation
• Restricted category definitions in current methods
• Need for laborious spatial inputs in segmentation

Innovation

• Text-prompted pathology image segmentation model
• Largest pathology dataset with 275k triples
• Enhances diagnostic interpretability via feature importance