Thermodynamic Isomorphism of Transformers: A Lagrangian Approach to Attention Dynamics

2026-02-09

AI Summary

This work addresses the lack of a unified physical theory underlying Transformer attention mechanisms by modeling attention as a physical system governed by the principle of least action. It constructs an intelligent Lagrangian on a Riemannian manifold induced by the Fisher information metric, integrating information geometry, statistical physics, and field theory. The study establishes, for the first time, a first law of information thermodynamics, interpreting inference and learning as mechanical work and chemical evolution, respectively, and explains scaling laws and grokking through phase transitions. It further provides a Goldstone boson field-theoretic interpretation of RoPE and proves that softmax corresponds uniquely to the thermodynamic equilibrium state minimizing the free energy of an information gas, thereby revealing the electrodynamical coupling nature of query-key interactions and offering a unified physical mechanism for emergent behaviors in Transformers.

Technology Category

• Machine Learning: Information Theory
• Search and Optimization: Learning to Search
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Although the Transformer architecture has revolutionized artificial intelligence, its underlying mechanisms remain largely heuristic and lack a unified physical theory. In this work, we propose a first-principles framework for information dynamics, treating the attention mechanism as a physical system governed by the principle of least action rather than as an algorithmic optimization. By mapping information states to a Riemannian manifold with the Fisher information metric, we derive the intelligence Lagrangian. We show that the softmax function corresponds to the unique thermodynamic equilibrium state that minimizes the Helmholtz free energy of the information gas. In addition, we identify the query-key interaction as an electrodynamic coupling between an external field and an intrinsic dipole moment. This theory establishes the first law of information thermodynamics, unifying inference (mechanical work) and learning (chemical evolution). It also explains emergent phenomena, such as scaling laws and grokking, as phase transitions characterized by the divergence of specific heat. Finally, we discuss how rotational symmetry breaking in the attention manifold generates massless Goldstone bosons, providing a field-theoretic perspective on rotary positional embeddings (RoPE). Our work connects Statistical Physics and Deep Learning, laying the groundwork for a general theory of physics-based intelligence.

Problem

• Transformer
• attention mechanism
• information thermodynamics
• physical theory
• first-principles

Innovation

• Thermodynamic Isomorphism
• Lagrangian Dynamics
• Information Thermodynamics
• Attention Mechanism
• Symmetry Breaking