Tensor Decomposition of Transformer Key-Value Caches: Spectral Structure and Format Comparison

2026-09-23

AI Summary

研究通过比较四种张量分解方法（Tucker、CP、张量链和t-SVD）处理自回归变换器的键值缓存，发现Tucker在保持全秩模式的同时提供最低重建误差。

Abstract

The key-value (KV) cache of autoregressive transformers can be viewed as a fourth-order tensor spanning attention heads, tokens, features, and grouped layers. We measure the singular-value spectra of all four mode unfoldings on Mistral-7B-v0.3 and LLaMA-2-13B and compare four standard tensor decompositions: Tucker, CP, tensor train, and t-SVD, at matched storage. The spectra partition the four axes into two classes. The token and feature modes carry low-rank structure, particularly for keys. The head and layer modes are nearly full-rank and resist compression at any practical error level. Among the four decompositions, Tucker achieves the lowest reconstruction error at every compression ratio from $2\times$ to $5\times$, because it can leave the full-rank modes untouched. Comparisons with two-dimensional unfolding baselines show that the preferred representation differs between keys and values: 2D methods achieve lower key error, while four-way Tucker achieves lower value error at matched storage. A mode-pinning theorem certifies the full-rank preservation from the measured spectra alone. Two further spectral properties affect the compressible modes without touching the full-rank ones: values reach a higher error floor than keys at every ratio, and post-RoPE keys lose $41\%$ - $64\%$ of their pre-RoPE compressibility on both models.

Problem

• Tensor Decomposition
• Key-Value Cache
• Singular Value Spectra
• Compression
• Autoregressive Transformers

Innovation

• Tensor Decomposition
• Tucker Decomposition
• Spectral Structure
• Key-Value Cache
• Compression Ratio