Massive Spikes in LLMs are Bias Vectors: Mechanistic Uncovering and Spike-Free Quantization

2026-06-01

AI Summary

This work addresses the detrimental impact of activation outliers in large language models (LLMs), which severely degrade quantization performance due to their expanded dynamic range. The study reveals, for the first time, that these outliers originate from structured vector biases in attention weight matrices rather than the commonly assumed scalar offsets. By analyzing the geometric alignment of weight projections and the rotational stability under RoPE perturbations, the authors propose INSERTQUANT, a spike-free post-training quantization framework. This approach integrates template-based vector reconstruction to preserve activation fidelity, achieving state-of-the-art tensor-level quantization accuracy on LLMs. Notably, INSERTQUANT demonstrates strong generalization beyond textual modalities, successfully extending to vision Transformers and other non-language architectures.

Abstract

Massive activation spikes in Large Language Models (LLMs) severely degrade quantization by stretching dynamic ranges. While prior hypotheses characterize these as high-level scalar biases, we argue that they are merely the scalar intermediates of rigid, structural vector biases in the spike-carrying tokens. We show that these tokens converge to constant vectors after normalization that drive the attention sink and value-state drain mechanisms. We geometrically substantiate this by analyzing the coordination of projection weights: $W_K$ contrastively amplifies the vector, $W_Q$ aligns semantic tokens toward it, and $W_V$ projects it into the spectral null-space. Furthermore, we reveal that the model actively preserves these structural biases against Rotary Positional Embedding (RoPE) perturbations by localizing them in "zones of rotational stability" utilizing low-frequency bands and coherent channel pairs. Leveraging this, we propose INSERTQUANT, a post-training quantization (PTQ) framework that clamps spikes and restores their function via pre-computed template vectors. This renders activations strictly spike-free, enabling robust low-bit quantization with high fidelity. INSERTQUANT achieves parity with state-of-the-art per-tensor quantization methods on LLMs and uniquely generalizes beyond text to other modalities such as ViTs.

Problem

• activation spikes
• quantization
• Large Language Models
• bias vectors
• dynamic range

Innovation

• spike-free quantization
• structural bias vectors
• rotational stability
• post-training quantization
• mechanistic interpretability