Generative Criticality in Large Language Model Temperature Scaling

2026-06-04

AI Summary

This work investigates whether text generation by large language models exhibits critical phenomena akin to those in physical systems across varying softmax temperatures. Treating token embeddings as a one-dimensional continuous spin chain, the study introduces, for the first time, a criticality analysis framework from statistical field theory—employing order parameters, correlation functions, susceptibility derived therefrom, and finite-size scaling collapse—into language model research. Across multiple scales (0.6B–32B) and prompt types in Qwen3, the authors observe a sharp peak in susceptibility, power-law correlations, abrupt changes in the order parameter, and condensation along semantic directions. Notably, the intrinsic dimensionality reaches a minimum at the critical temperature, revealing a temperature-tuned continuous-like phase transition accompanied by semantic condensation.

Abstract

We propose a statistical-field framework for text generated by large language models (LLMs), treating token embeddings as continuous spin variables on a one-dimensional chain. Defining a susceptibility from the connected two-point correlator and an order parameter from the ensemble-averaged embedding field, we vary the \texttt{softmax} temperature $T$ and observe a sharp susceptibility peak near a characteristic $T_c$ with power-law-like scaling, a concurrent rapid change in the order parameter, and a collapse onto a single semantic direction below $T_c$. The intrinsic dimension estimated by the two nearest neighbor (TwoNN) method independently corroborates these findings, reaching a minimum near $T_c$. Results are robust across model scales (Qwen3: 0.6B--32B) and prompt categories. While the phenomenology closely resembles a continuous phase transition, the non-equilibrium nature of autoregressive generation warrants further investigation. Our framework provides quantitative tools for probing the collective statistical structure of LLM outputs and suggests connections between decoding strategies and critical phenomena.

Problem

• criticality
• large language models
• temperature scaling
• phase transition
• statistical structure

Innovation

• generative criticality
• temperature scaling
• statistical field theory
• phase transition
• intrinsic dimension