SigmaScale: LLM Compression with SVD-based Low-Rank Decomposition and Learned Scaling Matrices

2026-06-05

AI Summary

Traditional singular value decomposition (SVD)-based compression of large language models often fails to preserve critical information, leading to significant performance degradation. This work proposes a data-driven, learnable row/column diagonal scaling mechanism that optimizes low-rank SVD decomposition under an activation-aware loss, replacing conventional analytical scaling with a more flexible approach tailored to the weight structure. The method effectively reduces the effective intrinsic rank of weight matrices, which exhibits strong correlation with compression loss. Experiments on Llama-3.1-8B-Instruct and Qwen3-8B demonstrate that the proposed approach matches state-of-the-art SVD-based compression methods in terms of perplexity and zero-shot performance while substantially lowering inference computational costs.

Abstract

We present SigmaScale, a method for learning auxiliary scaling matrices $S$ to aid truncated Singular Value Decomposition (SVD) based Large Language Model (LLM) compression. Instead of deriving scaling matrices analytically, SigmaScale optimizes two sets of vectors that define diagonal row and column scaling transformations under an activation-aware compression loss. We show that learned scaling lowers the effective intrinsic rank of weight matrices, as reflected by reductions in effective-rank entropy, and that this reduction is strongly correlated with compression loss. Experiments on Llama 3.1 8B Instruct and Qwen3-8B show that SigmaScale is competitive with closely related state-of-the-art SVD-based compression methods across perplexity and zero-shot benchmarks. By using learned activation-aware transformations, SigmaScale explores a more flexible route to low-rank LLM compression by adapting to the structure of individual model weights. The advantage observed in specific tasks makes our approach a valid option for applications requiring a reduced LLM-inference computing cost.

Problem

• LLM compression
• low-rank decomposition
• SVD
• scaling matrices
• effective rank

Innovation

• SVD-based compression
• learned scaling matrices
• activation-aware loss
• low-rank decomposition
• LLM compression