Variational Adapter for Cross-modal Similarity Representation

2026-05-29

AI Summary

Existing vision-language models often rely on coarse-grained binary matching annotations, which can introduce spurious negative samples and impair generalization. This work formulates image-text matching under fine-grained semantic scarcity as a variational inference problem for the first time. By introducing a variational adapter, the method constructs a probabilistic cross-modal similarity representation in the latent space and incorporates a regularization mechanism to mitigate overfitting to hard labels. This approach more faithfully captures uncertainty and consistently outperforms current methods across multiple benchmarks, including image-text retrieval, domain generalization, and generalization from base to novel classes, demonstrating superior robustness and generalization capability.

Abstract

The core of vision-language models lies in measuring cross-modal similarity within a unified representation space. However, most image-text matching or multi-class image classification datasets lack fine-grained cross-modal matching annotations, forcing the continuous similarity space into binary classification boundaries. This compression induces false negative samples and significantly impairs the generalization performance of cross-modal tasks. While prior research has attempted to mitigate this by modeling intra-modal ambiguity, it often overlooks inherent annotation flaws, leading to suboptimal uncertainty allocation. To address these challenges, we propose a Variational Adapter for Cross-modal Similarity Representation (VACSR). This approach reformulates image-text matching with fine-grained semantic scarcity as a variational inference problem. It constructs a latent space for cross-modal similarity and uses regularization techniques to mitigate overfitting to binary annotations. Experiments on image-text retrieval, domain generalization, and base-to-novel generalization demonstrate the proposed method's effectiveness and robust generalization ability.

Problem

• cross-modal similarity
• fine-grained annotation scarcity
• binary classification boundaries
• false negative samples
• generalization performance

Innovation

• variational inference
• cross-modal similarity
• fine-grained semantics
• annotation scarcity
• representation learning