Perceptual Anchoring: Prototype-Guided Text Calibration for Training-free Open-Vocabulary Semantic Segmentation

2026-08-04

AI Summary

This work addresses the semantic gap between generic text embeddings and instance-specific visual representations that plagues current training-free open-vocabulary semantic segmentation methods, often resulting in incomplete masks or erroneous segmentations. To bridge this gap, we propose Prototype-guided Text Calibration (PTC), the first approach to introduce a perception-anchoring mechanism into this task. Our method first constructs category-specific visual prototypes grounded in reliable visual evidence and then adaptively calibrates corresponding text embeddings using these prototypes, dynamically aligning generic semantics with instance appearance. Requiring no additional training or external models, PTC is plug-and-play and consistently enhances six state-of-the-art methods across eight benchmarks, yielding more complete and accurate segmentation masks, thereby demonstrating its effectiveness and broad applicability.

Abstract

Training-free open-vocabulary semantic segmentation (OVSS) partitions an image into semantically distinct regions based on arbitrary text descriptions, without learning any additional parameters. However, existing methods typically focus on improving visual representations while treating text embeddings that encode only generic category concepts as fixed classification references. The resulting semantic gap between these generic concepts and the visual representations that capture the specific appearances of target instances often causes incomplete masks and erroneous predictions in non-target regions. Inspired by the symbol-percept correspondence underlying perceptual anchoring, we propose Prototype-Guided Text Calibration (PTC) for training-free OVSS. In the Perceiving stage, PTC selects reliable visual evidence based on initial matching scores to construct category-specific visual prototypes. In the Anchoring stage, PTC uses these prototypes to calibrate their corresponding text embeddings, with the calibration strength adaptively adjusted based on the amount of visual evidence. Consequently, the calibrated text embeddings align more accurately with instance-specific visual representations while preserving generic category semantics and open-vocabulary generalization. Moreover, PTC requires neither additional training nor external models and can serve as a plug-and-play module for existing methods. Extensive experiments across eight benchmarks show that PTC significantly enhances the performance of six representative methods and yields more complete and accurate segmentation results. These results validate PTC as a simple and effective approach to improving visual-text alignment.

Problem

• open-vocabulary semantic segmentation
• semantic gap
• text-visual alignment
• training-free
• perceptual anchoring

Innovation

• Perceptual Anchoring
• Prototype-Guided Text Calibration
• Training-free
• Open-Vocabulary Semantic Segmentation
• Visual-Text Alignment