InfoNCE: Identifying the Gap Between Theory and Practice

2024-06-28

arXiv.org

AI Summary

This work addresses the theory-practice gap in contrastive learning: while classical InfoNCE theory assumes homoscedasticity or partial invariance of latent variables, real-world augmentations (e.g., aggressive cropping) induce anisotropic and continuously varying latents. To bridge this gap, we propose AnInfoNCE—a novel loss function that, for the first time, guarantees provable factor identifiability under anisotropic latent structures, substantially extending the theoretical applicability of contrastive learning. Grounded in an information-theoretic framework, we validate AnInfoNCE through identifiability analysis, controlled synthetic experiments, and empirical evaluation on CIFAR-10 and ImageNet. Results demonstrate that AnInfoNCE effectively recovers previously collapsed latent information. Moreover, our analysis uncovers an intrinsic trade-off between representation identifiability and downstream discriminative performance—highlighting a fundamental limitation in current contrastive paradigms.

Technology Category

• Machine Learning: Information Theory
• Search and Optimization: Learning to Search
• Computer Vision: Representation Learning for Vision

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Prior theory work on Contrastive Learning via the InfoNCE loss showed that, under certain assumptions, the learned representations recover the ground-truth latent factors. We argue that these theories overlook crucial aspects of how CL is deployed in practice. Specifically, they either assume equal variance across all latents or that certain latents are kept invariant. However, in practice, positive pairs are often generated using augmentations such as strong cropping to just a few pixels. Hence, a more realistic assumption is that all latent factors change with a continuum of variability across all factors. We introduce AnInfoNCE, a generalization of InfoNCE that can provably uncover the latent factors in this anisotropic setting, broadly generalizing previous identifiability results in CL. We validate our identifiability results in controlled experiments and show that AnInfoNCE increases the recovery of previously collapsed information in CIFAR10 and ImageNet, albeit at the cost of downstream accuracy. Finally, we discuss the remaining mismatches between theoretical assumptions and practical implementations.

Problem

• Bridging theory-practice gap in InfoNCE contrastive learning
• Addressing unrealistic latent factor assumptions in CL
• Proposing AnInfoNCE for anisotropic latent factor recovery

Innovation

• AnInfoNCE generalizes InfoNCE for anisotropic settings
• Validated via controlled experiments on CIFAR10/ImageNet
• Addresses latent factor variability in contrastive learning