supervised contrastive alignment

Existing two-stage cross-modal alignment methods suffer from suboptimal semantic alignment due to distributional mismatches across modalities. To address this, we propose the first single-stage, trimodal joint contrastive learning framework for end-to-end semantic alignment of audio, visual, and textual modalities. Our approach abandons the sequential alignment paradigm, instead constructing a unified representation space via cross-modal attention and multimodal embedding. A novel triplet loss is introduced to enhance contrastive learning across all three modalities simultaneously. Evaluated on the AVCaps dataset, our method achieves the first empirical validation of single-stage alignment superiority: audio-driven visual retrieval improves by 2× over two-stage baselines, and cross-modal retrieval consistently outperforms state-of-the-art two-stage models across all modalities. These results demonstrate both the effectiveness and scalability of unified multimodal representation learning.

Existing methods rely on a single fixed anchor modality to align multimodal data, leading to anchor sensitivity, insufficient intra-modal information exploitation, and failure to model inter-non-anchor-modality correlations. This paper proposes CentroBind, which abandons the fixed-anchor paradigm and constructs a unified multimodal representation space. We introduce an adaptive centroid-anchor mechanism: multimodal features are dynamically clustered to generate learnable centroids, and contrastive learning is jointly optimized with geometric constraints to simultaneously refine intra-modal representations, inter-modal associations, and cross-modal alignment. Theoretical analysis guarantees three key learning properties, effectively mitigating single-anchor dependency and information loss. On both synthetic and real-world benchmarks, CentroBind achieves average improvements of 3.2%–5.8% over state-of-the-art baselines—including ImageBind—across cross-modal retrieval and zero-shot classification tasks.

Multimodal representation learning faces challenges including anchor-modality dependency—leading to insufficient cross-modal alignment—and instability in singular value optimization. To address these, we propose an anchor-free multimodal joint alignment framework: (i) we design a softmax loss over dominant singular values as logits to enforce alignment along shared principal directions across modalities; and (ii) we introduce instance-level contrastive regularization to enhance class separability and training stability. Theoretical analysis grounded in singular value decomposition (SVD) characterizes structural properties of the learned representation matrices. Extensive experiments demonstrate significant improvements over state-of-the-art baselines on cross-modal retrieval and classification tasks, validating the method’s effectiveness, robustness, and generalization capability. The source code will be made publicly available.

Cross-modal retrieval suffers from semantic misalignment due to geometric inconsistency between image and text representations. This work formulates image–text matching as a metric alignment problem in the embedding space. We systematically demonstrate that: (1) the Wasserstein distance quantitatively characterizes inter-modal distributional discrepancy; (2) cosine similarity exhibits superior robustness over Euclidean distance, KL divergence, and other conventional metrics for alignment; and (3) standard MLPs inadequately capture complex cross-modal interactions. To address these issues, we propose a lightweight neural metric learning framework that integrates vision-language models with unimodal encoders, jointly optimizing four classical metrics and two learnable neural metrics. Our approach achieves significant improvements in Recall@K across standard cross-modal retrieval benchmarks. Moreover, it provides practical, deployment-oriented alignment evaluation criteria and architectural design guidelines.

Current multimodal large language models (MLLMs) struggle to achieve token-level semantic alignment between vision and language under image-level alignment, limiting the visual understanding and reasoning capabilities of small-scale LLMs. To address this, we propose the first supervised token-level embedding alignment mechanism: leveraging vision–language priors from pre-trained models (e.g., CLIP), we distill cross-modal alignment knowledge via contrastive learning and design a lightweight adapter to precisely map visual tokens into the LLM’s embedding space. Our method requires no additional training data or inference overhead. Evaluated on multiple visual question answering and reasoning benchmarks, it improves performance of small-scale MLLMs by 3.2–5.7 points, while significantly enhancing model interpretability and generalization.

This work proposes UniCon, a unified framework that addresses the inefficiency of small-batch stochastic optimization commonly used in contrastive learning. By reformulating the contrastive alignment problem into an analytically solvable form, UniCon introduces a contrastive similarity weighting matrix and derives a closed-form global solution in a reproducing kernel Hilbert space (RKHS), thereby eliminating the need for conventional backpropagation. The approach seamlessly accommodates both linear and nonlinear encoders and supports diverse alignment paradigms, while also uncovering a fundamental connection between contrastive learning and spectral methods. Empirical evaluations demonstrate that UniCon substantially improves training efficiency across synthetic, unimodal, multimodal, and zero-shot tasks without compromising—indeed, often enhancing—generalization performance.

This work addresses a critical limitation in existing independently trained multimodal contrastive models—such as CLIP, SigLIP, and FLAVA—which lack explicit alignment of their representation spaces, particularly in image–text coupling consistency. The authors theoretically demonstrate that embedding spaces from image and text encoders, despite being trained under different architectures and data distributions, can be simultaneously aligned via a single orthogonal mapping. Building on this insight, they propose a unified alignment framework integrating orthogonal mapping modeling, multimodal kernel consistency analysis, and anchor set validation. Notably, the method operates without requiring re-embedding, enabling seamless compatibility with pre-trained models. Extensive experiments across multiple established architectures validate its effectiveness, while also offering a novel perspective on privacy-preserving multimodal representation learning.

This work addresses the limitation of existing cross-modal alignment methods, which often conflate semantic and non-semantic information, leading to insufficient semantic consistency and alignment bias caused by modality gaps. To overcome this, we propose a semantic alignment framework based on constrained disentanglement and distribution sampling. Specifically, a dual-path UNet architecture adaptively disentangles visual and linguistic representations into semantic and modality-specific components, aligning only the extracted semantic factors. Furthermore, a multi-constraint optimization strategy combined with distribution-aware sampling is introduced to effectively bridge inter-modality discrepancies, thereby enhancing the reasonableness and robustness of alignment. Extensive experiments demonstrate that our method consistently outperforms state-of-the-art approaches across multiple benchmarks and backbone architectures, achieving performance gains of 6.6% to 14.2%.

Concept alignment lacks a unified definition, and existing methods optimize divergent objectives under the same terminology, obscuring its fundamental nature. This work formalizes its multidimensional structure by decomposing it along two axes—“alignment targets” and “alignment levels”—and identifies four distinct alignment properties, revealing that current approaches satisfy only subsets of these. To address this limitation, we propose Coupled Sparse Autoencoders (CoSAE), a framework that jointly optimizes multiple alignment objectives, alongside InterVenchA, an interventional evaluation benchmark. Experiments demonstrate that optimizing a single objective fails to reliably recover other alignment properties, whereas CoSAE achieves strong instance-level conceptual consistency using merely 0.1% paired data.

This work addresses the limitation of existing posterior multimodal alignment methods, which rely solely on global representations and thus struggle to support fine-grained cross-modal tasks under paired data scarcity. To overcome this, the authors propose a novel posterior alignment approach based on relative representations, introducing for the first time a token-level relative representation mechanism into posterior alignment. By incorporating lightweight, learnable anchors within each modality’s embedding space, the method models similarity relationships between image and text tokens, enabling fine-grained structural alignment. Notably, it avoids complex projection layers and achieves effective cross-modal fine-grained correspondence through anchor optimization alone. Extensive experiments demonstrate significant performance gains over state-of-the-art methods on zero-shot classification, cross-modal retrieval, and zero-shot segmentation, validating the effectiveness and generalizability of the proposed fine-grained alignment strategy.