Contextures: Representations from Contexts

2025-05-02

AI Summary

Current foundational models lack a systematic theoretical characterization of their learned representations. Method: We propose Contexture Theory, which formalizes mainstream representation learning as the approximation of the top-d singular functions of an expectation operator—induced by associations between inputs and contextual variables. Our framework unifies supervised, self-supervised, and manifold learning under a common representational mechanism; identifies contextual quality—not parameter count—as the fundamental determinant of diminishing returns in model scaling; integrates singular function analysis, expectation operator modeling, context utility quantification, and generalization-theoretic proofs; and introduces a downstream-task-agnostic metric for context quality assessment. Results: Empirical evaluation across multiple real-world datasets demonstrates that our context quality metric exhibits strong correlation with downstream task performance, establishing a new, interpretable, and quantitatively assessable paradigm for representation learning.

Technology Category

• Machine Learning: Representation Learning
• Computer Vision: Representation Learning for Vision
• Reasoning under Uncertainty: Uncertainty Representations

Application Category

• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Despite the empirical success of foundation models, we do not have a systematic characterization of the representations that these models learn. In this paper, we establish the contexture theory. It shows that a large class of representation learning methods can be characterized as learning from the association between the input and a context variable. Specifically, we show that many popular methods aim to approximate the top-d singular functions of the expectation operator induced by the context, in which case we say that the representation learns the contexture. We demonstrate the generality of the contexture theory by proving that representation learning within various learning paradigms -- supervised, self-supervised, and manifold learning -- can all be studied from such a perspective. We also prove that the representations that learn the contexture are optimal on those tasks that are compatible with the context. One important implication of the contexture theory is that once the model is large enough to approximate the top singular functions, further scaling up the model size yields diminishing returns. Therefore, scaling is not all we need, and further improvement requires better contexts. To this end, we study how to evaluate the usefulness of a context without knowing the downstream tasks. We propose a metric and show by experiments that it correlates well with the actual performance of the encoder on many real datasets.

Problem

• Characterizing representations learned by foundation models systematically
• Proving optimality of contexture-based representations in compatible tasks
• Evaluating context usefulness without knowing downstream tasks

Innovation

• Characterizes representations via context association
• Proves optimality of contexture-based representations
• Proposes metric for context usefulness evaluation