Few-Shot Concept Prompt Learning for Segmentation Foundation Models via Visual Grounding

2026-08-02

AI Summary

This work addresses the performance limitations of existing foundation models for medical image segmentation, which rely on natural language prompts but suffer from a scarcity of image–text paired data. The authors propose FS-CPL, a novel approach that introduces visually grounded continuous concept prompts to replace conventional textual prompts. Operating with a frozen backbone, FS-CPL requires only a few image–mask samples and optimizes learnable prompt embeddings through mask-supervised learning, eliminating the need for additional text data or backbone retraining. The method achieves substantial performance gains, improving Dice scores by up to 0.62 across four benchmarks—BUSI, HC18, TN3K, and CVC-Clinic—and demonstrates compatibility with both SAM3 and Medical SAM3, confirming its effectiveness and generalizability.

Abstract

Promptable segmentation foundation models (FMs) such as SAM3 and Medical SAM3 promise few-shot, interactively-specified segmentation for medical imaging through a natural language interface, yet their performance on clinical tasks falls well short of this promise. We posit that this shortfall is not an artefact of insufficient medical pretraining or imperfect prompt phrasing, but a structural limitation that will persist in any domain where paired image-text supervision is scarce, as it is across most clinical modalities. We further hypothesize that the limitation is specific to natural language as a control signal: a visually grounded prompt, learned directly from the target distribution, should recover the lost performance without additional image-text data or backbone retraining. We propose Few-Shot Concept Prompt Learning (FS-CPL), which learns a continuous concept prompt embedding $\mathbf{p}^* \in \mathbb{R}^{T \times d}$ from a small support set of $K$ image--mask pairs via mask supervision, with the encoder-decoder backbone frozen. Across four public benchmarks spanning ultrasound and endoscopy (BUSI, HC18, TN3K, CVC-Clinic), FS-CPL delivers absolute Dice improvements of up to $+0.62$ over canonical text prompts and is \emph{backbone-agnostic}: it lifts both vanilla SAM3 and the domain-specifically pretrained Medical SAM3, showing that visual concept prompting is complementary to in-domain pretraining.

Problem

• few-shot segmentation
• segmentation foundation models
• visual grounding
• medical imaging
• natural language prompting

Innovation

• Few-Shot Learning
• Prompt Learning
• Visual Grounding
• Segmentation Foundation Models
• Concept Prompt Embedding