🤖 AI Summary
This paper addresses the lack of theoretical grounding for singular value truncation thresholds in low-rank approximation of deep neural network (DNN) weight matrices. We propose a signal–noise decoupling framework grounded in Random Matrix Theory (RMT), modeling weights as the sum of a low-rank signal component and isotropic random noise, and derive an analytically justified denoising threshold. Furthermore, we introduce—novelty—the first threshold validity metric based on singular vector alignment, quantified as the cosine similarity between the estimated signal’s singular vectors and those of the original weight matrix; this advances beyond conventional empirical threshold selection relying solely on singular value spectra. Experiments across multiple mainstream DNN weight matrices demonstrate that our metric quantitatively distinguishes the signal-preserving capability of competing thresholding methods, leading to significantly improved stability and interpretability of model accuracy after low-rank compression.
📝 Abstract
This study evaluates thresholds for removing singular values from singular value decomposition-based low-rank approximations of deep neural network weight matrices. Each weight matrix is modeled as the sum of signal and noise matrices. The low-rank approximation is obtained by removing noise-related singular values using a threshold based on random matrix theory. To assess the adequacy of this threshold, we propose an evaluation metric based on the cosine similarity between the singular vectors of the signal and original weight matrices. The proposed metric is used in numerical experiments to compare two threshold estimation methods.