TraceCLIP: Recovering Local Semantics from Patch-to-CLS Contributions

2026-07-28

AI Summary

This work addresses the challenge of recovering local visual semantics from CLIP’s global representations to support dense vision–language tasks, without introducing additional training or external models. The authors propose a training-free framework that uncovers the implicit local semantic accessibility within CLIP by analyzing the contribution of image patches to the attention output of the [CLS] token. They further construct a semantic–geodesic topological gate to calibrate affinities among image patches, enabling high-quality dense semantic reconstruction. Evaluated on eight zero-shot semantic segmentation benchmarks, the method achieves consistent improvements of 1.3–4.5 percentage points in mean Intersection-over-Union (mIoU) over the current best training-free approaches, demonstrating its effectiveness and strong generalization capability.

Abstract

Dense vision-language understanding, including object localization, region recognition, and open-vocabulary semantic segmentation, requires associating language concepts with spatially grounded visual regions. CLIP provides a strong foundation for these tasks by learning a shared image-text embedding space from large-scale contrastive pre-training. However, its image-level objective aligns text with a CLS-derived global representation, leaving local vision-language correspondence only indirectly constrained. Existing methods either introduce additional supervision, external models, or task-specific adaptation, while training-free approaches mainly recover dense responses from existing patch features without examining where local semantics become most accessible within CLIP. We introduce TraceCLIP, a training-free framework that recovers latent patch-level semantic evidence by isolating the patch-specific terms written into the CLS attention output. TraceCLIP further converts contribution-derived semantic responses into a semantic-geodesic topology gate that calibrates final-layer patch affinity for dense feature reconstruction. Diagnostic experiments show that these contribution features exhibit strong local semantic discrimination and text-conditioned spatial alignment. On eight zero-shot semantic segmentation benchmarks, TraceCLIP achieves gains of 1.3 to 4.5 points in average mIoU over the strongest prior training-free methods across both backbones and background settings, without additional training, external vision foundation models, or region-level supervision. More broadly, these findings suggest that spatially localized semantics may remain accessible within the internal construction of globally aligned representations.

Problem

• dense vision-language understanding
• local semantics
• CLIP
• semantic segmentation
• patch-to-CLS alignment

Innovation

• TraceCLIP
• patch-to-CLS contribution
• training-free dense vision-language alignment
• semantic-geodesic topology
• zero-shot semantic segmentation