🤖 AI Summary
This study investigates the intrinsic relationship between the architectural properties of large language models and their compressibility, revealing fundamental limitations of conventional compression approaches. Through over 40 systematic experiments on GPT-2 and Mistral 7B—employing canonical correlation analysis (CCA), R² linearity metrics, KL divergence, and rotation-based quantization—the work identifies five key phenomena: variance does not equate to importance; module behavior depends on input distribution; reconstruction error exhibits a “wall effect”; deeper linear components enhance compressibility; and approximately 30% of token computations are redundant. Building on these insights, the authors propose a block-level functional replacement strategy that achieves 34× compression in the final layer of Mistral 7B with only a 1.71 increase in perplexity. However, multi-block replacement fails due to error accumulation, exposing inherent bottlenecks of static compression and challenging prevailing paradigms.
📝 Abstract
We present a systematic empirical study of transformer compression through over 40 experiments on GPT-2 (124M parameters) and Mistral 7B (7.24B parameters). Our analysis covers spectral compression, block-level function replacement, rotation-based quantization, activation geometry, and adaptive early exit.
We identify five structural properties relevant to compression. (1) Variance is not importance: high-variance activation directions are approximately 96 percent uncorrelated with predictive directions (measured via CCA), and projecting onto these subspaces preserves over 90 percent of variance while degrading perplexity. (2) Block linearity is conditional: transformer blocks are approximately linear (R^2 ~ 0.95 on GPT-2, 0.93 on Mistral block 31) only under the correct upstream distribution; modifying earlier blocks induces distribution shift that degrades downstream approximations. (3) The reconstruction wall: approaches that factor weights into quantized components amplify errors through cross-terms, making direct quantization strictly superior. (4) Linearity increases with depth: Mistral 7B exhibits a progression from R^2 = 0.17 (block 0) to R^2 = 0.93 (block 31), indicating a division between nonlinear feature construction and linear refinement. (5) Approximately 30 percent of tokens are computationally easy, confirmed via exit heads and KL divergence sensitivity.
We demonstrate that single-block linear replacement achieves 34x compression with a 1.71 perplexity increase on the final block of Mistral 7B, while multi-block replacement fails due to residual error accumulation and distribution shift. These findings suggest fundamental limits to static post-training compression and motivate adaptive, per-token computation as a more effective direction.