cross-modal alignment

Existing two-stage cross-modal alignment methods suffer from suboptimal semantic alignment due to distributional mismatches across modalities. To address this, we propose the first single-stage, trimodal joint contrastive learning framework for end-to-end semantic alignment of audio, visual, and textual modalities. Our approach abandons the sequential alignment paradigm, instead constructing a unified representation space via cross-modal attention and multimodal embedding. A novel triplet loss is introduced to enhance contrastive learning across all three modalities simultaneously. Evaluated on the AVCaps dataset, our method achieves the first empirical validation of single-stage alignment superiority: audio-driven visual retrieval improves by 2× over two-stage baselines, and cross-modal retrieval consistently outperforms state-of-the-art two-stage models across all modalities. These results demonstrate both the effectiveness and scalability of unified multimodal representation learning.

This work addresses the suboptimal performance in multimodal representation alignment caused by modality gaps and data scarcity. To this end, the authors propose a disentangled representation learning framework based on shared and modality-specific codebooks. Leveraging a compositional vector quantization mechanism, the method decomposes multimodal features into shared semantic components and modality-unique components, and employs a progressive alignment strategy to optimize the alignment space without requiring fully paired data. The unified shared codebook effectively bridges the modality gap, while the modality-specific codebooks mitigate dominant-modality bias, enabling more balanced multimodal fusion. The approach achieves state-of-the-art performance across classification and retrieval tasks spanning nine modalities, including text, images, video, and audio.

Cross-modal retrieval suffers from semantic misalignment due to geometric inconsistency between image and text representations. This work formulates image–text matching as a metric alignment problem in the embedding space. We systematically demonstrate that: (1) the Wasserstein distance quantitatively characterizes inter-modal distributional discrepancy; (2) cosine similarity exhibits superior robustness over Euclidean distance, KL divergence, and other conventional metrics for alignment; and (3) standard MLPs inadequately capture complex cross-modal interactions. To address these issues, we propose a lightweight neural metric learning framework that integrates vision-language models with unimodal encoders, jointly optimizing four classical metrics and two learnable neural metrics. Our approach achieves significant improvements in Recall@K across standard cross-modal retrieval benchmarks. Moreover, it provides practical, deployment-oriented alignment evaluation criteria and architectural design guidelines.

This work addresses unsupervised cross-modal transfer—training a model solely on labeled data from a single source modality to enable zero-shot inference on unseen target modalities. Methodologically, it formulates cross-modal alignment as an invertible problem and achieves transfer via unsupervised projection of source-modality representations into modality-specific subspaces, under the assumption that semantic classes in the latent space follow a Gaussian Mixture Model (GMM). Theoretically, it provides the first rigorous proof that perfect multi-modal alignment is attainable under mild, realistic conditions. Methodologically, it introduces the first GMM-structured latent-space paradigm for unsupervised cross-modal transfer, eliminating any reliance on target-modality labels. Experiments on synthetic multi-modal Gaussian data validate the theoretical analysis and demonstrate substantial improvements in cross-modal inference accuracy.

This work addresses the modality gap in multimodal representation learning induced by the InfoNCE objective, which manifests as a conflict between inter-modal alignment and uniformity, as well as intra-modal alignment inconsistencies. The paper proposes the first framework that decouples alignment and uniformity in multimodal learning, employing Hölder divergence–based alignment optimization alongside a dedicated uniformity loss to effectively mitigate these conflicts. Theoretically, the proposed objective is shown to serve as a valid proxy for the global Hölder divergence between multimodal distributions. Notably, the method requires no task-specific components and consistently improves performance across both discriminative tasks (e.g., retrieval) and generative tasks (e.g., UnCLIP), demonstrating its generality and effectiveness.

This work addresses the systematic geometric misalignment—commonly referred to as the modality gap—between visual and linguistic representations in multimodal large language models, which hinders effective alignment and scalability. The authors propose a theoretical framework that decomposes the modality gap within a fixed reference frame, thereby relaxing the isotropic assumption inherent in conventional contrastive learning. This enables, for the first time, statistical alignment using only large-scale unpaired data without requiring aligned image–text pairs. Building on this theory, they introduce ReAlign, a training-free three-step alignment strategy (Anchor-Trace-Centroid), and ReVision, a scalable pretraining paradigm. Experiments demonstrate that the proposed approach significantly improves representation alignment without relying on high-quality paired data, offering a novel pathway toward efficient scaling of multimodal models.

This paper addresses the cross-modal semantic gap in multimodal understanding through a unified alignment–translation–fusion–transfer framework. Methodologically: (1) a spatial reasoning BERT is introduced to map spatial language to 2D layouts; (2) a medical term spatial co-occurrence loss is designed to ground textual descriptions in 3D anatomical locations; (3) a structured text-to-knowledge graph fact linking benchmark with interpretability is established; and (4) a multi-stream feature fusion mechanism coupled with cross-modal knowledge distillation enables lightweight RGB-based action recognition. Key contributions include: the first spatial semantic alignment model, joint anatomical-spatial representation learning, a standardized, interpretable knowledge graph linking benchmark, and a novel unimodal distillation paradigm that achieves near-fused performance without multimodal inputs. Experiments demonstrate significant improvements across all tasks: the RGB-only model attains accuracy comparable to multimodal baselines while reducing computational overhead by over 60%.

This work addresses a critical limitation in existing independently trained multimodal contrastive models—such as CLIP, SigLIP, and FLAVA—which lack explicit alignment of their representation spaces, particularly in image–text coupling consistency. The authors theoretically demonstrate that embedding spaces from image and text encoders, despite being trained under different architectures and data distributions, can be simultaneously aligned via a single orthogonal mapping. Building on this insight, they propose a unified alignment framework integrating orthogonal mapping modeling, multimodal kernel consistency analysis, and anchor set validation. Notably, the method operates without requiring re-embedding, enabling seamless compatibility with pre-trained models. Extensive experiments across multiple established architectures validate its effectiveness, while also offering a novel perspective on privacy-preserving multimodal representation learning.

This work addresses the challenge of collaborative perception among heterogeneous agents, where modality isolation—stemming from the absence of co-occurring modalities in training data—exacerbates cross-modal domain gaps and impedes effective cooperation. To overcome this limitation, the authors propose CodeAlign, a novel framework that achieves efficient modality alignment without requiring spatially co-located supervision. CodeAlign introduces a feature-codebook-feature (FCF) translation mechanism coupled with codebook regularization to explicitly enforce cross-modal representation consistency, thereby constructing a compact yet expressive shared semantic space. Notably, the method eliminates reliance on spatially overlapping observations and attains state-of-the-art performance on both OPV2V and DAIR-V2X benchmarks, using only 8% of the parameters of prior approaches and reducing communication overhead by a factor of 1024.

This work addresses the limitation of existing cross-modal alignment methods, which often conflate semantic and non-semantic information, leading to insufficient semantic consistency and alignment bias caused by modality gaps. To overcome this, we propose a semantic alignment framework based on constrained disentanglement and distribution sampling. Specifically, a dual-path UNet architecture adaptively disentangles visual and linguistic representations into semantic and modality-specific components, aligning only the extracted semantic factors. Furthermore, a multi-constraint optimization strategy combined with distribution-aware sampling is introduced to effectively bridge inter-modality discrepancies, thereby enhancing the reasonableness and robustness of alignment. Extensive experiments demonstrate that our method consistently outperforms state-of-the-art approaches across multiple benchmarks and backbone architectures, achieving performance gains of 6.6% to 14.2%.