cross-modal contrastive learning

This paper systematically analyzes four core challenges in multimodal alignment and fusion—cross-modal misalignment, semantic modality gap, computational bottlenecks, and data noise/heterogeneity—that hinder model robustness, generalizability, and scalability. To address them, the authors propose a unified taxonomy covering 200+ studies, revealing an integrated “alignment–fusion” co-design paradigm. They further introduce three principled solution pathways: (1) noise-robust learning, (2) heterogeneous representation modeling, and (3) few-shot cross-modal transfer—unifying contrastive learning, cross-modal attention, latent-space alignment, graph neural networks, and differentiable architecture search. Extensive experiments on social media analysis, medical imaging, and sentiment recognition validate the efficacy of the framework. The work distills actionable design principles for scalable, robust, and generalizable multimodal learning, establishing a new benchmark for both theoretical research and industrial deployment.

Existing two-stage cross-modal alignment methods suffer from suboptimal semantic alignment due to distributional mismatches across modalities. To address this, we propose the first single-stage, trimodal joint contrastive learning framework for end-to-end semantic alignment of audio, visual, and textual modalities. Our approach abandons the sequential alignment paradigm, instead constructing a unified representation space via cross-modal attention and multimodal embedding. A novel triplet loss is introduced to enhance contrastive learning across all three modalities simultaneously. Evaluated on the AVCaps dataset, our method achieves the first empirical validation of single-stage alignment superiority: audio-driven visual retrieval improves by 2× over two-stage baselines, and cross-modal retrieval consistently outperforms state-of-the-art two-stage models across all modalities. These results demonstrate both the effectiveness and scalability of unified multimodal representation learning.

This paper addresses the challenge of implicit cross-modal alignment (e.g., audio–text) under unpaired data conditions, proposing a Bayesian inference–based contrastive learning theoretical framework. Theoretically, it establishes—for the first time—that directly contrasting embeddings from two unpaired modalities (bypassing intermediate modalities) recovers the likelihood ratio under mild assumptions, implying that contrastive representations inherently possess probabilistic alignment properties. Methodologically, the work integrates geometric analysis of contrastive representations, probabilistic graphical modeling, and multimodal embedding space alignment to empirically validate the validity boundaries of key assumptions. Contributions include: (1) the first Bayesian-interpretable theory for unpaired cross-modal contrastive learning; (2) a novel paradigm for pre-trained model transfer; and (3) significant improvements in zero-shot cross-modal retrieval and ambiguity-aware reinforcement learning policies.

Multimodal contrastive learning often captures only redundant, shared information across modalities, failing to model modality-unique and synergistic interactions. To address this, we propose CoMM, a framework that abandons explicit cross-modal feature alignment and instead maximizes mutual information among augmented representations within a unified multimodal embedding space—enabling end-to-end co-modeling. For the first time, we rigorously disentangle multimodal information into redundant, unique, and synergistic components from an information-theoretic perspective, and theoretically prove that mutual information maximization inherently balances these three components. The method is fully differentiable and requires no paired multimodal supervision. Controlled ablation studies validate the accuracy of our information disentanglement, and CoMM achieves state-of-the-art performance on seven real-world multimodal benchmarks.

Vision-language pretraining (VLP) models exhibit insufficient adversarial robustness in image-text joint tasks, particularly under cross-modal joint perturbations. This work proposes the first gradient-based multimodal adversarial attack method grounded in contrastive learning, capable of simultaneously generating imperceptible adversarial images and texts. Crucially, it introduces a novel joint optimization framework that unifies cross-modal (image-text) and intra-modal contrastive losses, thereby substantially enhancing the transferability of adversarial examples across model architectures in black-box settings. Evaluated on image-text retrieval and visual entailment tasks, the method significantly outperforms both unimodal and state-of-the-art multimodal attacks: it achieves an average transfer success rate improvement of 27.6% over existing approaches, demonstrating superior cross-architecture generalization and efficacy in practical adversarial scenarios.

This work addresses the modality gap in multimodal representation learning induced by the InfoNCE objective, which manifests as a conflict between inter-modal alignment and uniformity, as well as intra-modal alignment inconsistencies. The paper proposes the first framework that decouples alignment and uniformity in multimodal learning, employing Hölder divergence–based alignment optimization alongside a dedicated uniformity loss to effectively mitigate these conflicts. Theoretically, the proposed objective is shown to serve as a valid proxy for the global Hölder divergence between multimodal distributions. Notably, the method requires no task-specific components and consistently improves performance across both discriminative tasks (e.g., retrieval) and generative tasks (e.g., UnCLIP), demonstrating its generality and effectiveness.

This work addresses the persistent modality gap between image and text embeddings in CLIP-style dual-encoder contrastive learning, despite their projection into a shared space. The authors identify this issue as stemming from mode collapse of the InfoNCE loss under low temperature settings. To mitigate this, they propose xNCE, a novel contrastive learning approach that jointly incorporates both cross-modal and intra-modal negative pairs. This strategy effectively narrows the modality gap while preserving the discriminative geometric structure of the embedding space. Experimental results demonstrate that xNCE maintains competitive performance on image-text retrieval benchmarks such as MS-COCO and achieves significant improvements in zero-shot classification accuracy across multiple standard datasets.

This work addresses the limitation of existing posterior multimodal alignment methods, which rely solely on global representations and thus struggle to support fine-grained cross-modal tasks under paired data scarcity. To overcome this, the authors propose a novel posterior alignment approach based on relative representations, introducing for the first time a token-level relative representation mechanism into posterior alignment. By incorporating lightweight, learnable anchors within each modality’s embedding space, the method models similarity relationships between image and text tokens, enabling fine-grained structural alignment. Notably, it avoids complex projection layers and achieves effective cross-modal fine-grained correspondence through anchor optimization alone. Extensive experiments demonstrate significant performance gains over state-of-the-art methods on zero-shot classification, cross-modal retrieval, and zero-shot segmentation, validating the effectiveness and generalizability of the proposed fine-grained alignment strategy.

Existing approaches struggle to model high-order dependencies among more than two modalities and lack a unified principle for balancing information retention and compression. This work introduces the information bottleneck principle into arbitrary multimodal alignment for the first time, proposing a One-vs-All multimodal alignment framework. By optimizing each modality’s sufficiency and minimality with respect to all others, the method derives a computable contrastive lower bound and a minimality regularizer. It further integrates parameter-free geometry-aware projection and a distribution-dependent upper-bound regularizer to effectively capture high-order interactions and geometric structures. The proposed approach achieves consistently strong and state-of-the-art performance across diverse tasks, including classification, regression, modality-agnostic evaluation, and cross-modal retrieval.

This work addresses the challenge of learning unified cross-modal representations in image–text contrastive pretraining, where modality disconnection often hinders effective alignment. To overcome this limitation, the authors propose a lightweight fusion and multi-level alignment mechanism that operates during training: it leverages fine-grained image–text correspondences to enhance alignment and introduces a structured interaction module to mitigate early saturation in contrastive learning and improve training stability. Notably, this module is removed at inference time, preserving the efficiency of the dual-encoder architecture. Experimental results demonstrate that the proposed approach significantly outperforms strong baselines across image–text retrieval, classification, and multimodal benchmark tasks, effectively bridging the modality gap while maintaining both discriminative representation quality and inference efficiency.

This work addresses the reliance of conventional multimodal large language models on costly and hard-to-scale fully aligned multimodal data. The authors propose a two-stage framework that trains such models using only pairwise modality data. In the first stage, a shared latent space is constructed through within-modality reconstruction and pairwise contrastive learning. In the second stage, new modality encoders are integrated with a pretrained decoder to enable cross-modal transfer and generation. Theoretical analysis establishes conditions under which aligned representations can be achieved using only pairwise data, introducing inductive biases based on partial alignment and minimal latent norms to eliminate the need for complete joint multimodal observations. The approach successfully incorporates 3D point clouds and tactile modalities into a pretrained model, achieving strong cross-modal performance across three pairs of modalities.