Context Matters: Support Set Selection and Failure Detection for In-Context Medical Image Segmentation

2026-08-05

AI Summary

This work addresses the critical yet underexplored role of support set selection in context-based medical image segmentation without fine-tuning, where performance and reliability are highly sensitive to support sample choice and no mechanism exists to preemptively detect segmentation failures. The authors propose a DINOv3 visual similarity–guided strategy to replace random support sampling and introduce a Transformer-based binary classifier that predicts—using only query and support images—whether the resulting segmentation will fall below a specified IoU threshold, thereby enabling failure warning without generating the actual segmentation output. This study is the first to systematically treat the support set as a controllable variable for optimizing contextual segmentation performance and to incorporate a segmentation-free failure prediction mechanism, enhancing safety for clinical deployment. Experiments across four benchmarks and three imaging modalities demonstrate consistent superiority or parity over random sampling, with pronounced gains under small support set sizes; the failure predictor significantly outperforms random baselines in all settings.

Abstract

In-context learning (ICL) adapts medical image segmentation models to unseen structures and modalities without retraining by conditioning on a task-specific support set of image-mask exemplars. Because this support set is the model's only task-specific signal, its composition directly influences segmentation performance. In this work, we investigate the support set as a controllable determinant of ICL reliability. First, we compare random sampling against similarity-based selection, where exemplars are retrieved based on their visual similarity to the query image. Second, we train a transformer-based classifier to predict, from the query and support images alone, whether a segmentation will fall below a specified Intersection-over-Union (IoU) threshold. Using MultiverSeg with DINOv3 embeddings across four benchmarks and three imaging modalities, we show that similarity-based selection consistently matches or outperforms random sampling, with the largest gains at the smallest support set sizes. Furthermore, our classifier predicts segmentation failure above chance on all four benchmarks. Ultimately, these results demonstrate that the reliability of in-context segmentation can be both improved via informed support selection and anticipated before use, providing practical mechanisms for safer clinical deployment.

Problem

• in-context learning
• support set selection
• failure detection
• medical image segmentation
• reliability

Innovation

• in-context learning
• support set selection
• failure detection
• medical image segmentation
• similarity-based retrieval