Geometry of Human Perceptual Domains Emerges Transiently in LLM Representations

2026-05-27

AI Summary

This study investigates whether large language models trained exclusively on text can spontaneously develop geometric representational structures analogous to human perceptual experiences—such as color, pitch, emotion, and taste. By analyzing the intrinsic geometric properties of residual stream activations across layers in multiple open-source Transformer models, and integrating representational geometry analysis, inter-layer alignment metrics, and correlation assessments against human perceptual benchmarks, this work systematically demonstrates for the first time that perceptual geometry transiently yet robustly emerges in intermediate model layers. This emergence follows a dynamic trajectory characterized by early dispersion, mid-layer organization, and late-stage attenuation, with distinct temporal profiles across different perceptual domains. Crucially, the representational geometry at key intermediate layers exhibits strong alignment with human perceptual benchmarks.

Abstract

While large language models (LLMs) are trained purely on textual data, prior work has shown that their internal representations can exhibit rich geometric structure in embedding space. Building on this line of work, we investigate whether such structure is similar to human perceptual organisation across different domains (e.g., color, pitch, emotion, and taste). Specifically, we study the layer-wise emergence of intrinsic geometrical structure corresponding to perceptual modalities within the residual streams of multiple open-weight transformer architectures. Our results reveal three key findings. First, we observe the emergence of layer-wise geometric structure across multiple perceptual domains, despite the absence of any direct perceptual supervision during training. Second, these perceptual domains exhibit distinct emergence profiles, with both geometric structure and its alignment with human baselines following domain- and model-specific trajectories across depth. Third, this emergence follows a consistent representational trajectory: geometry is weak or diffuse in early layers, becomes progressively organised in intermediate layers, and is attenuated in later layers, suggesting that perceptual geometry arises transiently as part of the model's internal transformation pipeline. This provides new insight into how and where human-like perceptual geometry arises in LLMs, offering a principled pathway for mechanistic analysis of internal representations.

Problem

• perceptual geometry
• large language models
• embedding space
• human perception
• representational structure

Innovation

• perceptual geometry
• layer-wise emergence
• LLM representations
• intrinsic structure
• transient representation