Rethinking the Role of Tensor Decompositions in Post-Training LLM Compression

2026-06-02

AI Summary

This study addresses the lack of systematic evaluation of tensor decomposition methods for post-training compression of large language models (LLMs), particularly regarding their applicability to both dense and mixture-of-experts (MoE) architectures. Through a combination of theoretical analysis and empirical experiments, this work provides the first comprehensive assessment of tensor decomposition across diverse LLM architectures, revealing critical performance trade-offs. It identifies a fundamental mismatch between the shared subspace assumption inherent in tensor decomposition and the heterogeneous representations actually learned by modern LLMs. By delineating the practical boundaries and limitations of tensor-based compression in contemporary LLMs, this research offers actionable insights for efficient model deployment and releases its implementation code to facilitate further investigation.

Abstract

Post-training compression is essential for deploying large language models (LLMs) under tight resource constraints. Tensor decompositions have emerged as a promising direction, offering compact parameterizations well suited to Transformer weight structures. However, existing studies evaluate these methods in narrow settings, leaving unclear whether tensorization is effective at large-scale deployment. We systematically evaluate tensor compression across dense and MoE architectures, establishing performance trade-offs grounded in both empirical analysis and theoretical analysis. We identify a fundamental mismatch between the shared subspaces assumed by tensor decompositions and the heterogeneous representations learned by modern LLMs, thereby delineating their practical limits and clarifying their viable role in large-scale deployment. The code is available at https://github.com/brain-lab-research/TT-LLM.

Problem

• tensor decomposition
• post-training compression
• large language models
• heterogeneous representations
• model deployment

Innovation

• tensor decomposition
• post-training compression
• large language models
• heterogeneous representations
• MoE architectures