multimodal representation learning

Cross-modal retrieval suffers from semantic misalignment due to geometric inconsistency between image and text representations. This work formulates image–text matching as a metric alignment problem in the embedding space. We systematically demonstrate that: (1) the Wasserstein distance quantitatively characterizes inter-modal distributional discrepancy; (2) cosine similarity exhibits superior robustness over Euclidean distance, KL divergence, and other conventional metrics for alignment; and (3) standard MLPs inadequately capture complex cross-modal interactions. To address these issues, we propose a lightweight neural metric learning framework that integrates vision-language models with unimodal encoders, jointly optimizing four classical metrics and two learnable neural metrics. Our approach achieves significant improvements in Recall@K across standard cross-modal retrieval benchmarks. Moreover, it provides practical, deployment-oriented alignment evaluation criteria and architectural design guidelines.

Existing two-stage cross-modal alignment methods suffer from suboptimal semantic alignment due to distributional mismatches across modalities. To address this, we propose the first single-stage, trimodal joint contrastive learning framework for end-to-end semantic alignment of audio, visual, and textual modalities. Our approach abandons the sequential alignment paradigm, instead constructing a unified representation space via cross-modal attention and multimodal embedding. A novel triplet loss is introduced to enhance contrastive learning across all three modalities simultaneously. Evaluated on the AVCaps dataset, our method achieves the first empirical validation of single-stage alignment superiority: audio-driven visual retrieval improves by 2× over two-stage baselines, and cross-modal retrieval consistently outperforms state-of-the-art two-stage models across all modalities. These results demonstrate both the effectiveness and scalability of unified multimodal representation learning.

Conventional multimodal learning assumes explicit representation alignment is universally beneficial, yet its causal impact remains unverified. Method: This work systematically investigates the conditional effects of enforced alignment, proposing a controllable contrastive learning module that dynamically modulates alignment strength and establishes a quantitative relationship between alignment strength and inter-modal information redundancy—derived via information decomposition and synthetic data modeling. Contribution/Results: We demonstrate that alignment efficacy is not universal: strong alignment improves performance under high redundancy but harms modality-specific representation learning under low redundancy. To address this, we introduce a balanced mechanism that jointly preserves shared semantics and modality-specific characteristics. Empirical evaluation on both synthetic and real-world benchmarks confirms that this mechanism significantly enhances the generalization capability of unimodal encoders. Our core contribution is the formal establishment of “redundancy-dependent alignment”—a principled, interpretable, and tunable paradigm for multimodal representation learning.

Medical multimodal learning faces challenges including scarcity of paired data and reliance on proprietary or pretrained encoders. To address these, this paper proposes a single-encoder, parameter-sharing framework that unifies text and imaging modalities within a shared Transformer architecture. It introduces learnable modality embeddings for adaptive representation learning and designs a cross-modal parameter-sharing mechanism coupled with a joint contrastive alignment loss to alleviate low-resource generalization bottlenecks. Crucially, the approach eliminates modality-specific encoders. Evaluated across multiple medical multimodal benchmarks, it achieves significant improvements in few-shot settings (<1k samples): average retrieval accuracy increases by 4.2%, and classification F1 score improves by 3.8%. The core contribution is the first lightweight, parameter-shared multimodal representation learning paradigm explicitly designed for low-resource medical scenarios.

Existing approaches to enhancing the interpretability of multimodal unified representations rely on discretized representations but suffer from two key limitations: (1) Euclidean distance-based quantification ignores dimensional heterogeneity, inducing representation redundancy; and (2) uniform cross-modal alignment neglects modality-specific characteristics. To address these issues, we propose Training-Free Codebook Optimization (TOC) and Fine-Grained/Coarse-Grained Inter-Modal Information Decoupling (FCID)—the first framework enabling post-pretraining, gradient-free representation refinement and modality-adaptive information decoupling. TOC mitigates quantization redundancy via unsupervised codebook refinement, while FCID explicitly models modality-specific properties and disentangles shared versus private cross-modal information. Evaluated on cross-modal retrieval and zero-shot transfer tasks, our method achieves significant improvements over state-of-the-art baselines: representation redundancy is reduced by 37%, and modality specificity is enhanced by 21%.

This work addresses the unclear interaction between feature alignment and target fitting in cross-modal fine-tuning, which often leads to a mismatch between feature-label structures across source and target domains, thereby degrading generalization. For the first time, this study theoretically characterizes their relationship by introducing the notion of “feature-label distortion,” and establishes a provable generalization bound on target error. Based on this analysis, a principle for joint optimization of alignment and fitting is derived. The resulting framework offers interpretable and actionable design guidelines for cross-modal fine-tuning. Extensive experiments demonstrate that the proposed method significantly outperforms current state-of-the-art approaches across multiple benchmark datasets, confirming its effectiveness and broad applicability.

This work addresses the suboptimal performance in multimodal representation alignment caused by modality gaps and data scarcity. To this end, the authors propose a disentangled representation learning framework based on shared and modality-specific codebooks. Leveraging a compositional vector quantization mechanism, the method decomposes multimodal features into shared semantic components and modality-unique components, and employs a progressive alignment strategy to optimize the alignment space without requiring fully paired data. The unified shared codebook effectively bridges the modality gap, while the modality-specific codebooks mitigate dominant-modality bias, enabling more balanced multimodal fusion. The approach achieves state-of-the-art performance across classification and retrieval tasks spanning nine modalities, including text, images, video, and audio.

This work addresses the practical challenges in multimodal learning caused by missing or redundant modalities and the lack of theoretical understanding of how modality selection affects performance. It presents the first joint analysis of how the number of modalities and feature granularity influence generalization. By constructing a hierarchical structure of function classes corresponding to different modality subsets and leveraging pairwise complexity measures, the study derives generalization error bounds that quantify the discrepancy between the learned mapping and the true underlying mapping. The theoretical results demonstrate that fine-grained modality features effectively reduce hypothesis space complexity and enhance modality complementarity, thereby improving both convergence rates and prediction accuracy. These findings provide a rigorous theoretical foundation for designing effective multimodal learning systems.

Multimodal extension is often hindered by the high annotation cost of large-scale paired data, particularly in specialized domains such as medical imaging and molecular analysis. This work proposes TextME, a framework that, for the first time, maps diverse modalities—including images, audio, 3D, X-rays, and molecular data—into the embedding space of large language models using only textual descriptions, without any modality-paired supervision. By leveraging the geometric structure of pretrained contrastive encoders, TextME enables zero-shot cross-modal transfer purely through text-driven alignment. This approach establishes a novel paradigm for modality extension, achieving effective zero-shot retrieval across heterogeneous, unaligned modalities—such as audio-to-image or 3D-to-X-ray—while preserving the representational capacity of the pretrained encoders.

This paper addresses the cross-modal semantic gap in multimodal understanding through a unified alignment–translation–fusion–transfer framework. Methodologically: (1) a spatial reasoning BERT is introduced to map spatial language to 2D layouts; (2) a medical term spatial co-occurrence loss is designed to ground textual descriptions in 3D anatomical locations; (3) a structured text-to-knowledge graph fact linking benchmark with interpretability is established; and (4) a multi-stream feature fusion mechanism coupled with cross-modal knowledge distillation enables lightweight RGB-based action recognition. Key contributions include: the first spatial semantic alignment model, joint anatomical-spatial representation learning, a standardized, interpretable knowledge graph linking benchmark, and a novel unimodal distillation paradigm that achieves near-fused performance without multimodal inputs. Experiments demonstrate significant improvements across all tasks: the RGB-only model attains accuracy comparable to multimodal baselines while reducing computational overhead by over 60%.