Singular Vectors of Attention Heads Align with Features

2026-02-13

AI Summary

This study investigates whether singular vectors in attention mechanisms can reflect the internal feature representations of language models, addressing a gap in theoretical grounding within interpretability research. By integrating singular value decomposition, theoretical modeling, and sparse attention decomposition, the work provides the first theoretical justification for the alignment between singular vectors and model features, and derives operational criteria testable in real-world models. Empirical analysis in controlled-feature models confirms this alignment, while theoretical analysis characterizes the general conditions under which it holds. Furthermore, the study identifies sparse attention structures in actual language models that conform to these theoretical predictions, offering a novel perspective and a falsifiable explanatory framework for understanding attention mechanisms.

Technology Category

• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Identifying feature representations in language models is a central task in mechanistic interpretability. Several recent studies have made an implicit assumption that feature representations can be inferred in some cases from singular vectors of attention matrices. However, sound justification for this assumption is lacking. In this paper we address that question, asking: why and when do singular vectors align with features? First, we demonstrate that singular vectors robustly align with features in a model where features can be directly observed. We then show theoretically that such alignment is expected under a range of conditions. We close by asking how, operationally, alignment may be recognized in real models where feature representations are not directly observable. We identify sparse attention decomposition as a testable prediction of alignment, and show evidence that it emerges in a manner consistent with predictions in real models. Together these results suggest that alignment of singular vectors with features can be a sound and theoretically justified basis for feature identification in language models.

Problem

• singular vectors
• attention heads
• feature representations
• mechanistic interpretability
• language models

Innovation

• singular vectors
• attention heads
• feature alignment
• sparse attention decomposition
• mechanistic interpretability