Mask to Concept: Auto-Promptable SAM3 via Efficient Test-Time Concept Embedding Search for Few-Shot Annotation

2026-06-25

AI Summary

This work addresses the limitations of foundation models in medical image segmentation—namely their reliance on manual prompts, insufficient integration of fine-grained clinical knowledge, and low annotation efficiency—by proposing the M2C framework. Operating within a frozen SAM3 architecture, M2C automatically optimizes transferable visual concept embeddings from only a few annotated samples, enabling few-shot automatic annotation without additional modules or parameter fine-tuning. It further introduces a Hybrid Uncertainty Estimation (HUE) module that integrates active learning with a consistency check between concept-based and geometric prompts, establishing a human-in-the-loop self-reinforcing annotation cycle. Experiments demonstrate that M2C achieves state-of-the-art few-shot performance on medical segmentation benchmarks, substantially improving both annotation efficiency and accuracy while requiring minimal expert intervention and offering high scalability.

Abstract

Transforming foundation segmentation models from human-prompted tools into auto-promptable annotators is critical for scalable medical data annotation. Current methods commonly depend on external feature matchers or auxiliary networks to automate geometric prompting, but introducing architectural overhead and limiting performance scalability. Although SAM3 natively supports concept segmentation via reusable text prompts, its direct use in medical imaging is hindered by a lack of fine-grained clinical knowledge and the ambiguity of human-written descriptions. In this work, we propose Mask to Concept (M2C), an efficient framework that adapts SAM3 for medical few-shot annotation without external modules, parameter retraining, or manual text engineering. Using only a few labeled images, M2C enables SAM3 to automatically search for transferable visual concepts entirely within its frozen architecture: it initializes a learnable concept embedding, uses it to prompt segmentation, and updates the embedding by gradients of minimizing the concept segmentation error. We further introduce a Hybrid Uncertainty Estimation (HUE) module that calculates the prediction entropy and maps concept predictions back to the box prompts, measuring concept-geometry prompting inconsistency. Highly uncertain samples are flagged actively for human correction, and the corrected masks are then fed back to M2C to continuously search for more precise concept embeddings, forming a self-enhancing annotation loop with minimal expert effort. Experiments on medical segmentation benchmarks show that our method achieves SOTA few-shot segmentation performance and outstanding annotation efficiency, offering a practical and efficient pathway toward scalable medical image labeling. Codes are at https://github.com/Huster-Hq/M2C.

Problem

• few-shot annotation
• medical image segmentation
• auto-prompting
• concept embedding
• foundation models

Innovation

• auto-prompting
• concept embedding
• few-shot segmentation
• uncertainty estimation
• SAM3