I-Con: A Unifying Framework for Representation Learning

2025-04-23

AI Summary

Existing representation learning paradigms—such as clustering, spectral methods, dimensionality reduction, contrastive learning, and supervised learning—employ diverse, ad hoc loss functions lacking a unified theoretical foundation. Method: This paper proposes I-Con, an information-theoretic unifying framework that models these paradigms as minimizing the integral KL divergence between two conditional distributions. Contribution/Results: I-Con is the first to uncover a shared implicit information-geometric structure across paradigms, enabling principled cross-paradigm loss composition and yielding an interpretable debiasing principle. Its theory-driven construction ensures both interpretability and generalization. Empirically, I-Con improves unsupervised classification accuracy on ImageNet-1K by 8% over prior state-of-the-art, and significantly mitigates bias in contrastive representation learning.

Technology Category

• Machine Learning: Representation Learning
• Computer Vision: Representation Learning for Vision
• Game Theory and Economic Paradigms: Adversarial Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

As the field of representation learning grows, there has been a proliferation of different loss functions to solve different classes of problems. We introduce a single information-theoretic equation that generalizes a large collection of modern loss functions in machine learning. In particular, we introduce a framework that shows that several broad classes of machine learning methods are precisely minimizing an integrated KL divergence between two conditional distributions: the supervisory and learned representations. This viewpoint exposes a hidden information geometry underlying clustering, spectral methods, dimensionality reduction, contrastive learning, and supervised learning. This framework enables the development of new loss functions by combining successful techniques from across the literature. We not only present a wide array of proofs, connecting over 23 different approaches, but we also leverage these theoretical results to create state-of-the-art unsupervised image classifiers that achieve a +8% improvement over the prior state-of-the-art on unsupervised classification on ImageNet-1K. We also demonstrate that I-Con can be used to derive principled debiasing methods which improve contrastive representation learners.

Problem

• Unifying diverse loss functions in representation learning
• Generalizing modern machine learning loss functions
• Improving unsupervised classification and debiasing methods

Innovation

• Unified information-theoretic framework for representation learning
• Generalizes diverse loss functions via integrated KL divergence
• Enhances unsupervised classification and debiasing techniques