patch-wise semantic matching

This paper addresses instance-level image retrieval—precisely localizing images containing the same object despite significant variations in scale, pose, and appearance. We propose Patchify, a fine-tuning-free framework that partitions database images into structured local patches and performs cross-granularity matching between global query features and patch-level features, enabling spatially interpretable retrieval. We introduce LocScore, a novel localization-aware evaluation metric, and reveal the critical role of information preservation during feature compression; to this end, we integrate Product Quantization for efficient compression. Experiments demonstrate that Patchify consistently outperforms global-feature baselines across multiple benchmarks and backbone architectures, significantly improving re-ranking accuracy. The method supports real-time retrieval over databases of up to ten million images, achieving a favorable trade-off among high accuracy, spatial interpretability, and strong scalability.

Existing attention-based sparse image matching models exhibit significant performance variations across different local features, yet the individual contributions of detectors and descriptors remain unclear. This work systematically investigates this issue and reveals that the choice of detector has a far greater impact on matching performance than that of the descriptor. Building on this insight, we propose a general, detector-agnostic zero-shot matching strategy: by fine-tuning a Transformer-based matcher with keypoints aggregated from multiple pre-existing detectors, our approach eliminates the need for retraining on any specific detector. Experiments demonstrate that, in zero-shot settings, our method achieves matching accuracy on novel detectors that matches or even surpasses that of models explicitly trained for those detectors, confirming its effectiveness and strong generalization capability.

Semantic correspondence aims to match semantically consistent keypoints across images, yet has long suffered from the absence of systematic surveys and unified evaluation protocols. This paper introduces the first comprehensive, reproducible semantic correspondence benchmark. Our contributions are threefold: (1) We propose the first hierarchical taxonomy of methodologies, clarifying the technical evolution and conceptual relationships among existing approaches; (2) We design a lightweight, efficient baseline model that achieves state-of-the-art performance on major benchmarks including PF-PASCAL and SPair-71k; (3) We establish cross-dataset standardized evaluation protocols and perform component-wise ablation studies, substantially improving experimental comparability and reproducibility. All code, configuration files, and evaluation tools are publicly released under an open-source license, enabling plug-and-play reproduction.

Semantic image correspondence requires modeling long-range inter-image dependencies, yet conventional approaches struggle to capture complex structural relationships. State-of-the-art methods based on 4D correlation volumes suffer from prohibitive computational cost, limiting scalability to high-resolution inputs and large receptive fields. To address this, we propose MambaMatcher—the first method to introduce selective state space models (SSMs) into semantic correspondence. We design a similarity-aware selective scanning mechanism that efficiently models the 4D correlation volume in linear complexity, enabling global dependency learning over high-resolution features. Our approach breaks the accuracy-efficiency trade-off: it achieves state-of-the-art performance on standard benchmarks including SPair-71k and PF-PASCAL, significantly outperforming both feature-matching and correlation-based methods. This demonstrates the strong representational capacity and practical viability of SSMs for visual correspondence tasks.

To address the content-dependent nature of superpixel segmentation—which leads to irregular regions and poor matching robustness—this paper proposes SuperPatch, a stable region descriptor based on superpixel neighborhoods, and extends PatchMatch to SuperPatchMatch, incorporating spatial neighborhood structure to enhance cross-image region correspondence accuracy. Furthermore, we develop an end-to-end, database-driven fast annotation framework that integrates superpixel segmentation, randomized matching optimization, neighborhood graph modeling, and similarity-guided cross-image label propagation. Evaluated on facial landmark annotation and medical image segmentation tasks, our method achieves superior accuracy over contemporary state-of-the-art approaches while significantly reducing computational overhead.

This study addresses the performance stagnation of existing semantic correspondence methods under fine-grained thresholds, which stems from quantization errors induced by the discrete grids of Vision Transformers (ViTs). To overcome this limitation, this work proposes the concept of continuous feature fields, constructing a continuous feature space via implicit neural representations to support queries at arbitrary coordinates. Specifically, a FiLM-conditioned decoder is employed to embed sub-pixel positional information into ViT features, thereby transcending discrete grid constraints and enabling continuous coordinate sampling. The proposed method achieves a 6.2 percentage point improvement in PCK@0.01 on the SPair-71k benchmark, substantially enhancing fine-grained semantic matching accuracy.

This work addresses the challenge of generalizing image retrieval models to unseen domains when large-scale instance-level annotations are unavailable. The authors propose a novel cross-domain similarity computation method based on local descriptor correspondences, which innovatively operates in similarity space rather than representation space. By integrating optimal transport with a data-dependent gain mechanism to suppress spurious matches and aggregating image-level similarity through strong correspondence voting, the approach achieves both efficiency and interpretability. Evaluated on a new benchmark comprising eight cross-domain datasets, the method significantly outperforms existing techniques, demonstrating superior average performance in out-of-domain scenarios while substantially reducing computational overhead.

This work addresses the limitations of existing local feature methods that rely solely on appearance cues, resulting in unstable keypoints and poorly discriminative descriptors. To overcome this, the authors propose a multi-cue guided local feature learning framework that innovatively couples semantic segmentation with surface normal prediction and incorporates a depth-aware stability assessment. This leads to a Semantic–Depth-Aware Keypoint selection mechanism (SDAK) and a Unified Three-Cue Fusion descriptor module (UTCF). Built upon a lightweight backbone and a joint prediction head, the proposed approach significantly improves local feature detection and matching performance across four benchmark datasets, demonstrating the effectiveness of jointly modeling semantic, geometric, and depth cues.

Existing image similarity metrics such as LPIPS and CLIP often fail to align with human subjective judgments in text-to-image generation tasks, particularly in personalized or context-sensitive scenarios. This work proposes CLPIPS, which uniquely leverages user-provided ranking feedback on generated images to fine-tune the layer combination weights of LPIPS through a lightweight adaptation. By optimizing these weights using a margin-based ranking loss on human-annotated data, CLPIPS achieves personalized alignment with perceptual similarity. Consistency with human judgments is evaluated using Spearman’s rank correlation coefficient and intraclass correlation coefficient. Experimental results demonstrate that CLPIPS significantly outperforms the original LPIPS in capturing user preferences, thereby validating the efficacy of lightweight, personalized fine-tuning for perceptual similarity assessment.

Existing vision foundation models lack a unified, fine-grained evaluation protocol for structured object understanding, particularly suffering from inconsistent evaluation setups and insufficient supervision in part-level semantic correspondence tasks across instances and categories. This work proposes the SOCO benchmark, which establishes the first unified framework for semantic object correspondence, featuring million-scale functional keypoint annotations and accompanying textual descriptions across 100 object categories. Systematic evaluation of vision and vision-language foundation models reveals that visual backbones exhibit strong semantic structure awareness but limited cross-category generalization; large vision-language models outperform purely visual approaches in text-guided localization; and crucially, semantic correspondence performance serves as a more effective predictor than ImageNet accuracy for downstream tasks such as segmentation, tracking, and 3D pose estimation.