Training-Free Generalized Few-Shot Segmentation through Open-Vocabulary Semantic Arbitration

2026-06-08

AI Summary

This work addresses the limitation of generalized few-shot semantic segmentation (GFSS), which typically requires task-specific training to incorporate novel classes, by proposing Open-V—the first training-free framework for this setting. Open-V leverages frozen foundation models during inference, harmonizing their semantic priors through a pixel-wise semantic arbitration mechanism that integrates promptable segmentation from Segment Anything (SAM) with K-shot CLIP-based support set centroids. This enables zero-training generalization to unseen categories. Experimental results demonstrate that Open-V achieves base/novel/harmonic mIoU scores of 78.4/77.5/77.9 on PASCAL-5i under the 1-shot setting, surpassing the strongest training-based baseline by 17.7 harmonic mIoU points. The study also uncovers a significant bias in current foundation-model-based segmentation evaluation protocols.

Abstract

Generalized Few-Shot Semantic Segmentation (GFSS) has traditionally been approached as a representation-learning problem, requiring task-specific adaptation to incorporate novel classes from limited support examples. Recent foundation models, however, already exhibit strong open-vocabulary recognition and segmentation capabilities, raising a different question: can GFSS be solved through inference-time coordination of frozen semantic priors rather than parameter adaptation? We answer this question with Open-V, a training-free GFSS framework that combines Segment Anything (SAM3) Promptable Concept Segmentation (PCS) with a K-shot CLIP support centroid through calibrated per-pixel semantic arbitration. OpenV introduces no trainable components and supports arbitrary semantic categories at inference time. Beyond segmentation performance, our study contributes three broader findings. First, we show that support information can be incorporated through inference-time semantic grounding, and that its contribution increases as foundation-model text priors weaken on label-disjoint vocabularies. Second, we identify a reproducibility confound in foundationmodel segmentation, demonstrating that preprocessing and evaluation-space mismatches can silently distort reported performance. Finally, we validate Open-V across PASCAL5i, COCO-20i, and ADE-OW, showing that training-free coordination of foundation-model priors generalizes across both conventional GFSS and open-vocabulary evaluation settings. On PASCAL-5i (1-shot), Open-V attains base/novel/harmonic mIoU of 78.4/77.5/77.9, without GFSS-specific training surpassing the strongest trained baseline by +17.7 HM.

Problem

• Generalized Few-Shot Segmentation
• Open-Vocabulary
• Foundation Models
• Semantic Segmentation
• Training-Free

Innovation

• training-free
• open-vocabulary segmentation
• semantic arbitration
• foundation models
• generalized few-shot segmentation