cross-modal alignment

This study investigates the representational potential of foundation models for cross-modal alignment—specifically, whether their unimodal representations inherently capture task-specific semantics and exhibit cross-modal transferability. Method: We formalize “representational potential” and systematically analyze structural regularities and semantic consistency across vision, language, and speech foundation models. Leveraging cross-modal similarity metrics, representation visualization, and neuroscience-inspired evaluation protocols, we assess generalizability and unification capacity across diverse architectures. Contribution/Results: Empirical results demonstrate that pretrained foundation models implicitly acquire semantic invariances requisite for cross-modal alignment—even when trained on unimodal data—thereby exhibiting strong potential as unified multimodal representation backbones. Our work establishes a theoretical framework for cross-modal alignment and introduces a reproducible, multi-faceted evaluation paradigm grounded in representational analysis.

Cross-modal retrieval suffers from semantic misalignment due to geometric inconsistency between image and text representations. This work formulates image–text matching as a metric alignment problem in the embedding space. We systematically demonstrate that: (1) the Wasserstein distance quantitatively characterizes inter-modal distributional discrepancy; (2) cosine similarity exhibits superior robustness over Euclidean distance, KL divergence, and other conventional metrics for alignment; and (3) standard MLPs inadequately capture complex cross-modal interactions. To address these issues, we propose a lightweight neural metric learning framework that integrates vision-language models with unimodal encoders, jointly optimizing four classical metrics and two learnable neural metrics. Our approach achieves significant improvements in Recall@K across standard cross-modal retrieval benchmarks. Moreover, it provides practical, deployment-oriented alignment evaluation criteria and architectural design guidelines.

Existing two-stage cross-modal alignment methods suffer from suboptimal semantic alignment due to distributional mismatches across modalities. To address this, we propose the first single-stage, trimodal joint contrastive learning framework for end-to-end semantic alignment of audio, visual, and textual modalities. Our approach abandons the sequential alignment paradigm, instead constructing a unified representation space via cross-modal attention and multimodal embedding. A novel triplet loss is introduced to enhance contrastive learning across all three modalities simultaneously. Evaluated on the AVCaps dataset, our method achieves the first empirical validation of single-stage alignment superiority: audio-driven visual retrieval improves by 2× over two-stage baselines, and cross-modal retrieval consistently outperforms state-of-the-art two-stage models across all modalities. These results demonstrate both the effectiveness and scalability of unified multimodal representation learning.

This work addresses unsupervised cross-modal transfer—training a model solely on labeled data from a single source modality to enable zero-shot inference on unseen target modalities. Methodologically, it formulates cross-modal alignment as an invertible problem and achieves transfer via unsupervised projection of source-modality representations into modality-specific subspaces, under the assumption that semantic classes in the latent space follow a Gaussian Mixture Model (GMM). Theoretically, it provides the first rigorous proof that perfect multi-modal alignment is attainable under mild, realistic conditions. Methodologically, it introduces the first GMM-structured latent-space paradigm for unsupervised cross-modal transfer, eliminating any reliance on target-modality labels. Experiments on synthetic multi-modal Gaussian data validate the theoretical analysis and demonstrate substantial improvements in cross-modal inference accuracy.

Conventional multimodal learning assumes explicit representation alignment is universally beneficial, yet its causal impact remains unverified. Method: This work systematically investigates the conditional effects of enforced alignment, proposing a controllable contrastive learning module that dynamically modulates alignment strength and establishes a quantitative relationship between alignment strength and inter-modal information redundancy—derived via information decomposition and synthetic data modeling. Contribution/Results: We demonstrate that alignment efficacy is not universal: strong alignment improves performance under high redundancy but harms modality-specific representation learning under low redundancy. To address this, we introduce a balanced mechanism that jointly preserves shared semantics and modality-specific characteristics. Empirical evaluation on both synthetic and real-world benchmarks confirms that this mechanism significantly enhances the generalization capability of unimodal encoders. Our core contribution is the formal establishment of “redundancy-dependent alignment”—a principled, interpretable, and tunable paradigm for multimodal representation learning.

This work addresses the modality gap in multimodal representation learning induced by the InfoNCE objective, which manifests as a conflict between inter-modal alignment and uniformity, as well as intra-modal alignment inconsistencies. The paper proposes the first framework that decouples alignment and uniformity in multimodal learning, employing Hölder divergence–based alignment optimization alongside a dedicated uniformity loss to effectively mitigate these conflicts. Theoretically, the proposed objective is shown to serve as a valid proxy for the global Hölder divergence between multimodal distributions. Notably, the method requires no task-specific components and consistently improves performance across both discriminative tasks (e.g., retrieval) and generative tasks (e.g., UnCLIP), demonstrating its generality and effectiveness.

This work addresses the limitation of existing cross-modal alignment methods, which often conflate semantic and non-semantic information, leading to insufficient semantic consistency and alignment bias caused by modality gaps. To overcome this, we propose a semantic alignment framework based on constrained disentanglement and distribution sampling. Specifically, a dual-path UNet architecture adaptively disentangles visual and linguistic representations into semantic and modality-specific components, aligning only the extracted semantic factors. Furthermore, a multi-constraint optimization strategy combined with distribution-aware sampling is introduced to effectively bridge inter-modality discrepancies, thereby enhancing the reasonableness and robustness of alignment. Extensive experiments demonstrate that our method consistently outperforms state-of-the-art approaches across multiple benchmarks and backbone architectures, achieving performance gains of 6.6% to 14.2%.

This work addresses the limitation of existing posterior multimodal alignment methods, which rely solely on global representations and thus struggle to support fine-grained cross-modal tasks under paired data scarcity. To overcome this, the authors propose a novel posterior alignment approach based on relative representations, introducing for the first time a token-level relative representation mechanism into posterior alignment. By incorporating lightweight, learnable anchors within each modality’s embedding space, the method models similarity relationships between image and text tokens, enabling fine-grained structural alignment. Notably, it avoids complex projection layers and achieves effective cross-modal fine-grained correspondence through anchor optimization alone. Extensive experiments demonstrate significant performance gains over state-of-the-art methods on zero-shot classification, cross-modal retrieval, and zero-shot segmentation, validating the effectiveness and generalizability of the proposed fine-grained alignment strategy.

This work addresses the instability in multimodal sentiment analysis caused by representation misalignment when using independently pretrained modality encoders, which often fail to surpass strong text-only baselines. To overcome this limitation, the authors propose a multimodal framework centered on explicit alignment: visual inputs are first transformed into structured textual descriptions via a vision-language model, thereby unifying heterogeneous modalities into a shared linguistic space. Cross-modal alignment is further enhanced through semantic token selection and batch uniformity regularization, prioritizing alignment over complex fusion mechanisms. The proposed approach significantly outperforms existing unimodal and multimodal methods across multiple sentiment and emotion benchmarks, demonstrating the effectiveness and robustness of an alignment-first strategy.

This work addresses the suboptimal performance in multimodal representation alignment caused by modality gaps and data scarcity. To this end, the authors propose a disentangled representation learning framework based on shared and modality-specific codebooks. Leveraging a compositional vector quantization mechanism, the method decomposes multimodal features into shared semantic components and modality-unique components, and employs a progressive alignment strategy to optimize the alignment space without requiring fully paired data. The unified shared codebook effectively bridges the modality gap, while the modality-specific codebooks mitigate dominant-modality bias, enabling more balanced multimodal fusion. The approach achieves state-of-the-art performance across classification and retrieval tasks spanning nine modalities, including text, images, video, and audio.

This study addresses the challenge of unsupervised cross-modal alignment between independently trained models, which typically requires extensive paired data. Grounded in the Platonic Representation Hypothesis, this work proposes a pairing-free alignment method that establishes cross-modal correspondences by exploiting geometric similarities within representation spaces. Specifically, orthogonal mappings are estimated via the Wasserstein Procrustes algorithm combined with geometric initialization techniques to effectively align embedding spaces. Notably, this research provides the first demonstration that coarse-grained cross-modal alignment can be achieved using geometric initialization alone. Extensive experiments across multiple datasets and modalities reveal that the proposed approach significantly outperforms existing baselines under extremely limited paired-data scenarios. Furthermore, its practical utility is validated through successful application to text-to-image generation tasks.