🤖 AI Summary
This work addresses the weak theoretical foundation of self-supervised contrastive learning by modeling its essence from the perspective of approximating supervised representation learning objectives—revealing that contrastive losses such as InfoNCE asymptotically approximate ideal supervised targets (e.g., classifier separability). To bridge this gap, we propose two key innovations: (1) prototype-aware representation bias, which explicitly constrains the representation space to align with class-prototype structures; and (2) balanced contrastive loss, which explicitly modulates gradient contributions from positive and negative pairs to mitigate sampling bias. Our theoretical analysis establishes a rigorous connection between self-supervised and supervised learning, clarifying the interplay between positive and negative sample interactions. Extensive experiments demonstrate that the proposed method significantly improves representation quality and downstream task performance across multiple benchmarks.
📝 Abstract
Self-supervised representation learning has achieved impressive empirical success, yet its theoretical understanding remains limited. In this work, we provide a theoretical perspective by formulating self-supervised representation learning as an approximation to supervised representation learning objectives. Based on this formulation, we derive a loss function closely related to popular contrastive losses such as InfoNCE, offering insight into their underlying principles. Our derivation naturally introduces the concepts of prototype representation bias and a balanced contrastive loss, which help explain and improve the behavior of self-supervised learning algorithms. We further show how components of our theoretical framework correspond to established practices in contrastive learning. Finally, we empirically validate the effect of balancing positive and negative pair interactions. All theoretical proofs are provided in the appendix, and our code is included in the supplementary material.