EOVSAM: Efficient Open-Vocabulary Segmentation with SAM 3 in One Pass

2026-08-03

AI Summary

This work addresses the high computational cost incurred by exhaustive large-vocabulary traversal in open-vocabulary segmentation by proposing a single-stage, end-to-end framework built upon SAM³. By integrating de-prompt conditioning and an attention aggregation mechanism, the method enables efficient mask generation without relying on multi-stage pipelines or post-hoc heuristic processing. It supports open-vocabulary segmentation from arbitrary textual descriptions and effectively mitigates the closed-set collapse problem. Experimental results demonstrate that the model surpasses the original SAM³ in segmentation accuracy across multiple benchmarks, achieves up to a 338× speedup in inference, and maintains high accuracy even at low input resolutions.

Abstract

Open-vocabulary segmentation identifies and segments objects from arbitrary textual descriptions. SAM 3 supports noun-phrase-guided segmentation and achieves competitive open-vocabulary performance through exhaustive vocabulary traversal, yet suffers from prohibitive computational overhead as target categories scale. In this paper, we propose an Efficient Open-Vocabulary segmentation framework with SAM 3 (EOVSAM), which adapts SAM 3 for single-pass prediction. EOVSAM removes prompt conditioning to turn SAM 3 into an efficient mask generator and introduces a new Attentional Aggregation strategy to optimize open-vocabulary classification end-to-end. This formulation avoids the multi-stage pipelines and post-processing heuristics commonly used by existing methods, while mitigating the closed-set collapse that can arise when classification is optimized directly. EOVSAM consistently improves segmentation accuracy over vanilla SAM 3 on all evaluated datasets and accelerates inference by up to 338$\times$. Furthermore, EOVSAM maintains high accuracy at lower resolutions while achieving even more remarkable inference speeds. Experiments on standard semantic and panoptic segmentation benchmarks show that EOVSAM combines competitive or state-of-the-art accuracy with a substantial speed advantage over existing open-vocabulary segmentation models. Code and models are available at https://github.com/hustvl/EOVSAM.

Problem

• open-vocabulary segmentation
• computational overhead
• inference efficiency
• scalability

Innovation

• open-vocabulary segmentation
• SAM 3
• one-pass inference
• attentional aggregation
• efficient mask generation