SOCO: Benchmarking Semantic Object Correspondence in Vision Foundation Models

2026-05-29

AI Summary

Existing vision foundation models lack a unified, fine-grained evaluation protocol for structured object understanding, particularly suffering from inconsistent evaluation setups and insufficient supervision in part-level semantic correspondence tasks across instances and categories. This work proposes the SOCO benchmark, which establishes the first unified framework for semantic object correspondence, featuring million-scale functional keypoint annotations and accompanying textual descriptions across 100 object categories. Systematic evaluation of vision and vision-language foundation models reveals that visual backbones exhibit strong semantic structure awareness but limited cross-category generalization; large vision-language models outperform purely visual approaches in text-guided localization; and crucially, semantic correspondence performance serves as a more effective predictor than ImageNet accuracy for downstream tasks such as segmentation, tracking, and 3D pose estimation.

Abstract

Measuring structured object understanding in vision foundation models remains challenging due to inconsistent evaluation protocols and limited part-level supervision. Semantic correspondence (SC) evaluates this capability by testing whether object parts can be matched across instances and categories under large variations in appearance, viewpoint, and geometry. To enable a systematic SC evaluation, we introduce SOCO, a new benchmark for Semantic Object Correspondence that introduces a taxonomy of correspondence types and provides consistent, functionally meaningful keypoint annotations across 100 categories and over 1M correspondence pairs. In addition, SOCO includes keypoint language descriptions, enabling the evaluation of large vision-language models (LVLMs) and their fine-grained part-level understanding. Comprehensive experiments reveal that (i) vision foundation backbones encode strong semantic structure but transfer correspondences poorly across related categories and only partially capture object-part position, (ii) LVLMs are stronger at text-prompted part localization than at visual-reference cross-image matching, exposing a gap between language-grounded localization and fine-grained visual correspondence, and (iii) correspondence performance predicts performance on dense downstream tasks, including segmentation, tracking, 3D pose estimation, and 3D detection, more strongly than ImageNet classification. Together, these findings position SOCO as a benchmark for structured, part-level representation quality in vision and multimodal foundation models.

Problem

• Semantic Correspondence
• Vision Foundation Models
• Object Understanding
• Keypoint Annotation
• Benchmarking

Innovation

• Semantic Correspondence
• Vision Foundation Models
• Keypoint Annotation
• Vision-Language Models
• Structured Representation