Massive Activation Gating Channel in Large Language Models

2026-10-05

AI Summary

This study investigates the mechanisms underlying massive activation phenomena in large language models (LLMs). Methodologically, it employs a first-localization technique to identify fixed gating channels within the embedding layer and examines their interactions with spiking feed-forward networks. Theoretically, it proposes a quadratic-form output induction mechanism based on down-projection matrix mixing to elucidate how these channels govern massive activations. Experiments across six LLMs of varying scales confirm the ubiquity of such gating channels and their dominant role in shaping activation patterns. Overall, this work provides novel theoretical insights and empirical evidence for understanding anomalous activation behaviors within LLMs.

Abstract

Massive activations, a phenomenon in which a small number of hidden channels exhibit exceptionally large magnitudes, are pervasive in large language models (LLMs). However, the mechanism by which a token develops massive activations as it propagates through a pretrained LLM remains poorly understood. In this paper, we find that the emergence of massive activations is controlled by a single channel in the input embedding to a spike feed-forward network (FFN). The position of this channel is fixed for a particular LLM. We name this channel the massive activation gating channel (MAGC). When the value of the MAGC is sufficiently large (or small, depending on the LLM), the output of the spike FFN exhibits massive activations. Examining six LLMs across four model families and different model sizes, we verify the existence and effect of MAGC. We further provide a theoretical explanation of the mechanism by which MAGC induces massive activations. When the value of MAGC is sufficiently large (or small), the output of a spike FFN asymptotically reduces to a quadratic form that mixes a few columns of the down-projection matrix of the FFN. Since these columns exhibit the shape of massive activations, the output therefore exhibits massive activations.

Problem

• Massive Activations
• Large Language Models
• Activation Mechanism
• Feed-Forward Network

Innovation

• Massive Activations
• Gating Channel
• Feed-Forward Network
• Large Language Models
• Quadratic Form