Isotropy Cliffs: The Geometric Signature of Decision-Making in Large Language Models

2026-08-01

AI Summary

This study investigates the geometric properties of decision-making processes in large language models (LLMs) on multiple-choice question answering tasks, with a focus on isotropy dynamics and their relationship to critical decision layers. Through isotropy metrics, representational similarity analysis, and clustering-based visualizations, the authors systematically examine the evolution of intermediate-layer activations across several open-source models. They identify, for the first time, an “isotropy cliff”—a sharp transition in isotropy—as a universal geometric signature of successful model decisions, demonstrating its robustness to prompt perturbations. Experimental results show that the layer at which this transition occurs is strongly correlated with downstream task accuracy (r ≈ 0.84) and exhibits consistent stability and generalizability across five prominent models and multiple datasets.

Abstract

We investigate the geometry of decision-making in Multiple Choice Question Answering (MCQA) through the lens of isotropy. Analyzing five open-weight models across diverse datasets, we identify decision-critical transition layers characterized by a shift in isotropy, coinciding with a major representational change and the emergence of task-relevant clusters. We demonstrate that this synchronized geometric behavior is strongly correlated with downstream accuracy ($r\approx0.84$), displaying its relevance for successful decision-making. Furthermore, we show that this transition is robust to prompt variations, suggesting that it reflects a general mechanism of model behavior.

Problem

• isotropy
• decision-making
• large language models
• Multiple Choice Question Answering
• representation geometry

Innovation

• isotropy
• decision-making
• large language models
• geometric signature
• representation transition