contrastive learning

This work investigates the representation-theoretic properties of multimodal contrastive learning under nonlinear, non-Gaussian data distributions, focusing on how such methods can transcend predefined vector-dimensional constraints to adaptively discover the data’s intrinsic low-dimensional structure. We propose a temperature-optimized multimodal contrastive learning framework and theoretically establish that—under mild assumptions—it simultaneously maximizes inter-modal mutual information and automatically identifies and compresses representations to the true intrinsic dimension of the shared latent variable. This constitutes the first theoretical characterization linking the learned representation dimension in contrastive learning to the underlying data manifold dimension. Experiments on synthetic benchmarks and real-world multimodal datasets (e.g., CC3M, Kinetics) demonstrate that the resulting representations are both low-dimensional and highly informative, effectively bridging the gap between theoretical analysis and empirical performance.

This work addresses the weak theoretical foundation of self-supervised contrastive learning by modeling its essence from the perspective of approximating supervised representation learning objectives—revealing that contrastive losses such as InfoNCE asymptotically approximate ideal supervised targets (e.g., classifier separability). To bridge this gap, we propose two key innovations: (1) prototype-aware representation bias, which explicitly constrains the representation space to align with class-prototype structures; and (2) balanced contrastive loss, which explicitly modulates gradient contributions from positive and negative pairs to mitigate sampling bias. Our theoretical analysis establishes a rigorous connection between self-supervised and supervised learning, clarifying the interplay between positive and negative sample interactions. Extensive experiments demonstrate that the proposed method significantly improves representation quality and downstream task performance across multiple benchmarks.

Multimodal contrastive learning, widely adopted for representation alignment, often fails to achieve semantic consistency because standard contrastive losses maximize mutual information without suppressing modality-specific information. Method: We introduce the information bottleneck principle into multimodal alignment for the first time, proposing a differentiable variational regularizer that explicitly enforces modality-invariant representations and suppresses modality-specific features within the contrastive learning framework. Contribution/Results: Our method requires no additional annotations and significantly improves alignment accuracy and semantic consistency in controlled ablation studies and cross-modal retrieval tasks. Empirical results demonstrate both the effectiveness and generalizability of information-bottleneck-driven regularization for multimodal representation learning.

This work addresses the theory-practice gap in contrastive learning: while classical InfoNCE theory assumes homoscedasticity or partial invariance of latent variables, real-world augmentations (e.g., aggressive cropping) induce anisotropic and continuously varying latents. To bridge this gap, we propose AnInfoNCE—a novel loss function that, for the first time, guarantees provable factor identifiability under anisotropic latent structures, substantially extending the theoretical applicability of contrastive learning. Grounded in an information-theoretic framework, we validate AnInfoNCE through identifiability analysis, controlled synthetic experiments, and empirical evaluation on CIFAR-10 and ImageNet. Results demonstrate that AnInfoNCE effectively recovers previously collapsed latent information. Moreover, our analysis uncovers an intrinsic trade-off between representation identifiability and downstream discriminative performance—highlighting a fundamental limitation in current contrastive paradigms.

Domain shift arising from scanner heterogeneity in medical imaging undermines conventional contrastive learning, which relies on predefined image augmentations incapable of modeling realistic clinical domain variations—thereby limiting representation robustness and downstream generalizability. To address this, we propose a counterfactual contrastive learning framework—the first to integrate causal inference into contrastive representation learning. Our method employs causal image synthesis to generate semantically consistent positive pairs exhibiting authentic domain variations (e.g., vendor-specific scanner characteristics), precisely emulating clinically relevant shifts. It jointly optimizes SimCLR and DINO-v2 contrastive objectives. Evaluated across five multi-center chest X-ray and mammography datasets, our approach significantly improves robustness to domain shift. Downstream task performance surpasses standard contrastive baselines, particularly enhancing generalization on underrepresented scanner types and reducing performance disparities across sex-stratified subgroups.

Concept alignment lacks a unified definition, and existing methods optimize divergent objectives under the same terminology, obscuring its fundamental nature. This work formalizes its multidimensional structure by decomposing it along two axes—“alignment targets” and “alignment levels”—and identifies four distinct alignment properties, revealing that current approaches satisfy only subsets of these. To address this limitation, we propose Coupled Sparse Autoencoders (CoSAE), a framework that jointly optimizes multiple alignment objectives, alongside InterVenchA, an interventional evaluation benchmark. Experiments demonstrate that optimizing a single objective fails to reliably recover other alignment properties, whereas CoSAE achieves strong instance-level conceptual consistency using merely 0.1% paired data.

Contrastive learning excels in self-supervised representation learning, yet its theoretical analyses typically assume identical class distributions between pretraining and downstream tasks—failing to address real-world distribution shifts or novel label spaces, thereby limiting domain generalization. Method: We propose the first unified framework modeling two types of mismatch: (i) distribution shift within a shared latent class space, and (ii) emergence of entirely unseen labels at downstream evaluation. By integrating statistical divergence measures with contrastive representation theory, we derive a novel generalization error upper bound that jointly quantifies the impact of both distribution shift and label-space expansion. Results: Our bound provides the first theoretical guarantee for cross-domain classification under distributional and label-space mismatch. Crucially, we prove that contrastive representations retain a provable performance lower bound even under unseen class distributions—significantly extending the theoretical applicability of contrastive learning to domain generalization scenarios.

This work addresses the limitations of conventional multimodal representation learning, which relies on a shared-private dichotomy and struggles to capture latent factors shared only among subsets of modalities, often leading to excessive alignment of irrelevant signals and loss of complementary information. To overcome this, the authors propose a Hierarchical Contrastive Learning (HCL) framework that introduces, for the first time, a hierarchical latent variable structure to explicitly model globally shared, partially shared, and modality-specific components. A structure-aware contrastive objective is designed to align only those factors that are genuinely shared. Theoretical analysis establishes identifiability of the model without requiring correlation assumptions and provides recovery guarantees for the loading matrices along with bounds on prediction risk. Experiments demonstrate that HCL accurately recovers the hierarchical structure, effectively selects task-relevant components, and significantly improves representation quality and downstream prediction performance on multimodal electronic health records.

This work investigates under what positive sample sampling conditions contrastive learning can recover a meaningful geometric structure in the latent space. By constructing a measure-theoretic framework, the study introduces a “diversity condition” as a necessary requirement for the identifiability of latent geometry and elucidates the joint influence of sampling support and encoder inductive bias on representation identifiability. Theoretically, it is shown that under full-support sampling, the global optimum of InfoNCE recovers the latent structure up to orthogonal equivalence; however, under non-full support, non-orthogonal mappings may yield better solutions. To address this, the authors propose a support-corrected variant of InfoNCE and model representations using the von Mises–Fisher distribution, empirically validating on both synthetic and real-world data the critical role of inductive bias when sampling diversity is limited.

This work addresses a critical limitation in existing large-scale multimodal contrastive learning methods: the neglect of the intrinsic spectral structure of embedding features, which concentrates semantic information in a few dominant subspaces while leaving other dimensions vulnerable to noise and spurious correlations, thereby impairing generalization. To remedy this, we propose the Spectral Disentanglement and Enhancement (SDE) framework—the first to integrate spectral analysis into contrastive learning. SDE adaptively partitions features via singular value decomposition into strong-signal, weak-signal, and noise subspaces, and employs a curriculum-based spectral augmentation strategy to amplify informative components. Furthermore, it introduces a dual-domain contrastive loss operating in both feature and spectral domains to jointly optimize representation alignment and spectral regularization. Evaluated on major multimodal benchmarks, SDE significantly outperforms state-of-the-art methods, enhancing robustness and generalization while seamlessly integrating into existing contrastive learning pipelines.