🤖 AI Summary
Dimensional collapse in the embedding space remains a critical challenge in semi-supervised contrastive learning, degrading representation discriminability.
Method: We propose CLOP (Contrastive Learning with Orthogonal Prototypes), a novel loss function that geometrically regularizes the embedding structure by enforcing class prototypes to span orthogonal linear subspaces—thereby fundamentally mitigating dimensional collapse. We first identify and quantify the critical learning rate threshold at which standard contrastive loss induces collapse, and leverage this insight to design an orthogonal-prototype-driven semi-supervised objective. CLOP integrates contrastive learning, orthogonal constraint optimization, and geometric embedding-space regularization.
Results: CLOP achieves significant performance gains on both image classification and object detection benchmarks. Crucially, it exhibits strong robustness to variations in learning rate and batch size—addressing key practical limitations of existing contrastive methods.
📝 Abstract
Contrastive learning has emerged as a powerful method in deep learning, excelling at learning effective representations through contrasting samples from different distributions. However, dimensional collapse, where embeddings converge into a lower-dimensional space, poses a significant challenge, especially in semi-supervised and self-supervised setups. In this paper, we first identify a critical learning-rate threshold, beyond which standard contrastive losses converge to collapsed solutions. Building on these insights, we propose CLOP, a novel semi-supervised loss function designed to prevent dimensional collapse by promoting the formation of orthogonal linear subspaces among class embeddings. Through extensive experiments on real and synthetic datasets, we demonstrate that CLOP improves performance in image classification and object detection tasks while also exhibiting greater stability across different learning rates and batch sizes.