How Attention Sinks Emerge in Large Language Models: An Interpretability Perspective

2026-02-04

arXiv.org

AI Summary

This study investigates why position zero (P0) commonly serves as an attention sink in large language models and elucidates its underlying mechanism. Through mechanistic interpretability analyses and from-scratch pretraining experiments, the work demonstrates for the first time that the P0 attention sink arises from a purely structural property of causal attention, independent of semantic content, and identifies a P0-Sink subnetwork driven by this architectural bias. Building on this insight, the authors propose two parameter-free acceleration methods that significantly promote the early emergence of the attention sink. Experimental results show that these methods match the performance of Gated Attention in enhancing both pretraining efficiency and downstream task effectiveness, while consistently outperforming standard Transformers.

Abstract

Large Language Models (LLMs) often allocate disproportionate attention to specific tokens, a phenomenon commonly referred to as the attention sink. While such sinks are generally considered detrimental, prior studies have identified a notable exception: the model's consistent emphasis on the first token of the input sequence. This structural bias can influence a wide range of downstream applications and warrants careful consideration. Despite its prevalence, the precise mechanisms underlying the emergence and persistence of attention sinks remain poorly understood. In this work, we trace the formation of attention sinks around the first token of the input. We identify a simple mechanism, referred to as the P0 Sink Circuit, that enables the model to recognize token at position zero and induce an attention sink within two transformer blocks, without relying on any semantic information. This mechanism serves as the basis for the attention sink on position zero. Furthermore, by analyzing training traces from a 30B A3B MoE model trained from scratch, we find that this mechanism emerges early in training and becomes increasingly concentrated in the first two layers, suggesting a possible signal for tracking pre training convergence states.

Problem

• position zero
• attention sinks
• causal LLMs
• Transformer
• mechanistic interpretation

Innovation

• attention sinks
• position zero
• mechanistic interpretability
• causal attention
• parameter-free acceleration