KODA: Contrastive Representation Comparison and Alignment for Vision-Language Foundation Models

2026-06-02

AI Summary

Existing vision-language foundation models, such as CLIP and SigLIP, exhibit representational discrepancies that lack a structured explanation, and evaluating them solely through downstream task performance fails to reveal their underlying mechanisms. This work proposes KODA, a novel framework that formalizes contrastive embedding clustering as a constrained optimization problem. By constructing a unified multimodal kernel, KODA preserves coherent structure in target representations while suppressing coherence in reference representations, thereby identifying subsets of samples with distinctive contrastive clustering properties. The method integrates efficient approximation techniques—including modality-level kernel composition, kernel-based alignment, random Fourier features, and random projections—to enable scalable computation. Experiments demonstrate that KODA uncovers consistent, interpretable representational differences across models and generates effective sample subsets for representation alignment.

Abstract

Vision-language foundation models such as CLIP and SigLIP provide widely used representations for multimodal learning systems. While these models are typically compared through downstream performance, such evaluations often do not explain how their representations differ structurally. In this work, we study this problem through the task of Contrastive Embedding Clustering: identifying sample subsets that are weakly clustered under one representation but strongly clustered under another. We propose \emph{Kernel Optimization for Discrepancy Analysis (KODA)}, a kernel-based framework for contrastive representation comparison and alignment. KODA constructs unified multimodal kernels through modality-wise kernel composition and formulates discrepancy discovery as a constrained optimization problem that searches for coherent structures in one representation while suppressing coherence in a reference representation. This yields interpretable discrepancy directions associated with specific sample subsets and modality interactions. To scale KODA to large vision-language datasets, we develop randomized low-dimensional approximations of joint kernels using random projections, including Random Fourier Features for shift-invariant kernels. Empirically, KODA identifies consistent and interpretable discrepancy structures across vision-language representations and provides sample subsets for representation alignment. The code is available at https://github.com/yokiwuuu/KODA.

Problem

• vision-language foundation models
• representation comparison
• structural discrepancy
• contrastive embedding clustering
• multimodal representation

Innovation

• Contrastive Embedding Clustering
• Kernel Optimization
• Representation Discrepancy
• Multimodal Kernels
• Random Fourier Features