On the Relationship Between the Choice of Representation and In-Context Learning

2025-10-09

AI Summary

This work investigates the relationship between label representation—e.g., the semantic form of class labels—and learning capability—i.e., whether increasing the number of in-context examples consistently improves performance—in in-context learning (ICL). We propose a controllable label enumeration algorithm that systematically varies semantic relatedness among class labels and conduct comprehensive ICL experiments across multiple language model scales. Our results reveal an orthogonality between representation quality and sample scalability: label representation determines baseline performance, whereas the number of demonstrations independently and stably enhances performance; crucially, the relative ranking of different representations remains invariant across varying demonstration counts. This is the first empirical demonstration of the decoupling between representation selection and sample expansion effects in ICL. The findings provide foundational theoretical insights into ICL mechanisms and actionable guidance for prompt engineering and label design.

Technology Category

• Machine Learning: Representation Learning
• Computer Vision: Representation Learning for Vision
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

In-context learning (ICL) is the ability of a large language model (LLM) to learn a new task from a few demonstrations presented as part of the context. Past studies have attributed a large portion of the success of ICL to the way these in-context demonstrations are represented, particularly to how labels are represented in classification tasks. On the other hand, observations of the learning capacity of ICL (i.e., the extent to which more in-context demonstrations can lead to higher performance) have been mixed, and ICL is often thought to occur only under specific conditions. The interaction between these two aspects in ICL, representation and learning, has not been studied in depth until now. We hypothesize that they are largely independent of one another, such that the representation of demonstrations determines the baseline accuracy of ICL, while learning from additional demonstrations improves only on top of this baseline. We validate this hypothesis by developing an optimization algorithm that can enumerate a spectrum of possible label sets (representations) varying in semantic relevance. We then perform ICL with varying numbers of in-context demonstrations for each of these label sets. We observed that learning happens regardless of the quality of the label set itself, although its efficiency, measured by the slope of improvement over in-context demonstrations, is conditioned on both the label set quality and the parameter count of the underlying language model. Despite the emergence of learning, the relative quality (accuracy) of the choice of a label set (representation) is largely maintained throughout learning, confirming our hypothesis and implying their orthogonality. Our work reveals a previously underexplored aspect of ICL: the independent effects of learning from demonstrations and their representations on ICL performance.

Problem

• Investigating how label representation affects in-context learning baseline accuracy
• Examining whether learning from demonstrations depends on representation quality
• Validating independence between representation choice and learning capacity in ICL

Innovation

• Optimized algorithm enumerates label sets by semantic relevance
• Learning occurs regardless of label set quality in ICL
• Representation quality and learning efficiency are largely independent