Autonomy-of-Heads: Data-Free Sparse Attention from Frozen Query-Key Geometry

2026-08-07

AI Summary

This work addresses the computational and memory bottlenecks in long-context large language model inference, which stem from the quadratic complexity of attention mechanisms and the substantial overhead of KV caching. The authors propose a data-agnostic sparsification method that, for the first time, functionally categorizes attention heads based on the spectral effective rank of their weight matrices, distinguishing between retrieval-oriented and streaming heads to enable pre-execution sparse attention configuration. By analyzing the spectral properties of query-key projection kernel matrices, they devise a low-dimensional, efficient computation strategy that circumvents the construction of full-sized attention matrices. Experiments demonstrate that at 50% sparsity, the approach preserves 96.5% of full attention performance on average, reduces prefill and decoding latency by up to 41.4% and 66.0%, respectively, and cuts KV cache memory usage by 50.0% in 256K-token contexts.

Abstract

Long-context LLM inference is bottlenecked by quadratic attention computation and growing KV-cache costs. Existing sparse attention and KV-compression methods typically decide which tokens or heads to preserve from runtime attention scores, observation windows, calibration prompts, or learned gates, making head diagnosis input-dependent and costly to deploy. We propose Autonomy-of-Heads (AoH), a data-free method that identifies retrieval and streaming heads from the spectral geometry of query-key projections. AoH defines the kernel attention operator $M_h = W_K^{h\top}W_Q^h$ and uses its effective-rank as a weight-space measure of head function: concentrated spectra indicate a small number of dominant query-key matching directions and are associated with retrieval heads, whereas diffuse spectra indicate the absence of a dominant global matching direction and are associated with streaming heads. We further derive an efficient $d_\text{head}$-dimensional computation that avoids constructing the full $d_\text{model}\times d_\text{model}$ matrix. We conducted extensive experiments across models demonstrating that at 50\% sparsity, AoH retains 96.5\% of Full Attention performance on average while reducing prefill and decode latency by up to 41.4\% and 66.0\%, respectively, and KV-cache memory by 50.0\% at 256K tokens.

Problem

• long-context LLM inference
• quadratic attention computation
• KV-cache costs
• sparse attention
• input-dependent head selection

Innovation

• sparse attention
• data-free
• spectral geometry
• KV-cache compression
• effective rank