🤖 AI Summary
In-context learning (ICL) in large language models—enabling generalization without fine-tuning via prompting—lacks an interpretable, first-principles mechanism, particularly one grounded in fundamental physical laws.
Method: We establish the first analytical mapping from linear-attention Transformers to real-valued spin-glass systems, constructing a statistically rigorous theoretical framework rooted in equilibrium statistical physics.
Contribution/Results: We prove that task diversity drives the system’s Boltzmann distribution to converge uniquely to the correct weight configuration, thereby revealing the thermodynamic nature of ICL. This framework unifies the explanation for how pre-trained models spontaneously acquire predictive capability on novel prompts, providing the first physically grounded foundation for emergent behavior in large models—directly derived from spin-glass theory—and bridging deep learning with statistical physics.
📝 Abstract
Large language models show a surprising in-context learning ability -- being able to use a prompt to form a prediction for a query, yet without additional training, in stark contrast to old-fashioned supervised learning. Providing a mechanistic interpretation and linking the empirical phenomenon to physics are thus challenging and remain unsolved. We study a simple yet expressive transformer with linear attention and map this structure to a spin glass model with real-valued spins, where the couplings and fields explain the intrinsic disorder in data. The spin glass model explains how the weight parameters interact with each other during pre-training, and further clarifies why an unseen function can be predicted by providing only a prompt yet without further training. Our theory reveals that for single-instance learning, increasing the task diversity leads to the emergence of in-context learning, by allowing the Boltzmann distribution to converge to a unique correct solution of weight parameters. Therefore the pre-trained transformer displays a prediction power in a novel prompt setting. The proposed analytically tractable model thus offers a promising avenue for thinking about how to interpret many intriguing but puzzling properties of large language models.