Density-Aware Translation of Spurious Correlations in Zero-Shot VLMs

2026-06-01

AI Summary

Zero-shot vision-language models such as CLIP are susceptible to spurious correlations, where contextual cues overshadow semantic content, leading to unreliable classification. This work proposes Density-Aware Translation (DAT), which leverages—for the first time—the local geometric density characteristics of the anisotropic shell structure in CLIP’s embedding space. DAT constructs a relative density metric using class reference sets to recalibrate image-text similarity scores without fine-tuning or prompt engineering, thereby suppressing overconfident predictions in sparse regions while preserving semantically consistent dense matches. Experiments demonstrate that DAT significantly improves both worst-group and average accuracy across multiple benchmarks, effectively mitigating the amplification of spurious correlations caused by the modality gap and offering a reliable calibration mechanism for zero-shot multimodal classification.

Abstract

Vision-Language models (VLMs), such as CLIP, achieve powerful zero-shot classification. However, their predictions remain sensitive to spurious correlations, where contextual cues dominate over semantic content. Earlier solutions typically rely on fine-tuning or prompt engineering, which either undermine the advantages of pre-trained models or are prone to hallucination. In this work, we propose Density-Aware Translation (DAT) that refines image-text similarity scores using a local geometric density term derived from group reference sets. Our approach is motivated by the phenomenon that CLIP embeddings exhibit a modality gap and lie on an anisotropic shell in the feature space: common patterns cluster near the mean, while rare patterns are pushed outward. This geometry creates uneven alignment, where spurious correlations are amplified while semantically meaningful but rare cues are marginalised. To address this, we employ a relative measure to rescale similarities based on embedding density, suppressing overconfident scores in diffuse regions while preserving dense, semantically consistent matches. Experimental results on benchmark datasets demonstrate consistent improvements in worst-group and average accuracy, highlighting density-aware translation as a simple and effective calibration mechanism for reliable zero-shot classification using multimodal models.

Problem

• spurious correlations
• zero-shot VLMs
• modality gap
• anisotropic embedding
• semantic alignment

Innovation

• Density-Aware Translation
• spurious correlations
• zero-shot VLMs
• embedding density
• modality gap