What are you sinking? A geometric approach on attention sink

2025-08-04

AI Summary

This work investigates the fundamental nature of Attention Sinks (AS): whether they are mere architectural byproducts or reflect geometric principles underlying the construction of stable coordinate systems in high-dimensional representation spaces. Method: We systematically analyze attention maps across diverse Transformer architectures and conduct ablation studies—particularly on positional encoding schemes—to characterize AS from a geometric reference frame perspective. Contribution/Results: We demonstrate that AS spontaneously emerge early in training and constitute an optimal solution for establishing a stable geometric reference frame in high-dimensional space. We identify and categorize three canonical AS structures: centralized, distributed, and bidirectional. Our findings establish the universality and functional necessity of AS, revealing their role in underpinning the intrinsic stability of attention mechanisms. Moreover, this geometric interpretation provides principled, geometry-driven guidance for model design—including positional encoding strategies and special token placement—thereby bridging representational geometry with architectural engineering.

Technology Category

• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Machine Learning: Deep Neural Architectures and Foundation Models
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Attention sink (AS) is a consistent pattern in transformer attention maps where certain tokens (often special tokens or positional anchors) disproportionately attract attention from other tokens. We show that in transformers, AS is not an architectural artifact, but it is the manifestation of a fundamental geometric principle: the establishment of reference frames that anchor representational spaces. We analyze several architectures and identify three distinct reference frame types, centralized, distributed, and bidirectional, that correlate with the attention sink phenomenon. We show that they emerge during the earliest stages of training as optimal solutions to the problem of establishing stable coordinate systems in high-dimensional spaces. We show the influence of architecture components, particularly position encoding implementations, on the specific type of reference frame. This perspective transforms our understanding of transformer attention mechanisms and provides insights for both architecture design and the relationship with AS.

Problem

• Analyzes attention sink patterns in transformer attention maps
• Identifies geometric principles behind reference frame establishment
• Explores impact of architecture components on reference frame types

Innovation

• Geometric approach to analyze attention sink
• Identify three distinct reference frame types
• Study position encoding impact on reference frames