singular value decomposition

This work investigates the impact of matrix concatenation operations on singular value spectra, aiming to bridge a theoretical gap in the structural stability of SVDs under concatenation. Addressing the central question—“How are the singular values of a concatenated matrix determined by those of its constituent submatrices?”—we extend Weyl’s inequality to block-wise concatenation for the first time, establishing a quantitative analytical framework grounded in matrix perturbation theory and norm inequalities. We derive computable upper bounds on singular value deviations and rigorously prove that dominant singular values remain stable when the operator norms of the submatrices are comparable. This result provides theoretical guarantees and principled design guidance for low-rank approximation, robust matrix clustering, and compression algorithms.

This paper addresses the weak theoretical foundations of matrix decomposition in machine learning by systematically constructing a self-consistent, comprehensive, and modern-application-oriented pedagogical framework. Methodologically, it grounds the exposition in numerical linear algebra and matrix analysis, unifying classical decompositions—including LU, QR, SVD, and block triangular factorizations—while integrating numerical stability analysis and Hermitian/Hilbert space theory. Crucially, it bridges traditional numerical analysis with deep learning’s backpropagation setting, emphasizing differentiability and computational robustness of decompositions in algorithm design and model optimization. The primary contribution is a compact, dual-purpose (teaching and research) knowledge system that fills critical gaps in both theoretical coherence and machine-learning relevance present in existing literature, thereby providing rigorous mathematical foundations for high-dimensional data modeling and efficient training.

Existing asymmetric kernel singular value decomposition (KSVD) methods rely on finite-dimensional approximations, limiting their ability to handle infinite-dimensional feature maps, and their variational objectives may be unbounded. Method: We propose the Coupled Covariance Eigenproblem (CCE) framework—the first rigorous variational formulation of KSVD in infinite-dimensional Hilbert spaces—unifying asymmetric KSVD with covariance operator theory and accommodating arbitrary non-Mercer, asymmetric kernels. We further derive an asymmetric Nyström method based on coupled adjoint eigenfunctions, overcoming classical limitations of symmetric kernel approximation or linear SVD modeling. Contribution/Results: Experiments demonstrate that our method significantly outperforms symmetric-baseline and linear-SVD approaches across multiple tasks, achieving faster training convergence and improved generalization. This work provides the first empirical validation of the practical utility of asymmetric kernel learning.

This work addresses the issue of uncontrolled reconstruction errors in SVD-based compression of large collections of matrices when heuristic grouping is employed prior to concatenation. To overcome this limitation, the authors propose a theory-driven compressive clustering framework grounded in spectral analysis of horizontally concatenated matrices. They establish, for the first time, a globally provable upper bound on SVD reconstruction error and derive two novel spectral bounds based on a lower bound for singular value growth. Building upon these theoretical guarantees, they design three clustering algorithms with explicit error control, integrated with incremental approximate SVD to efficiently estimate compression error without explicitly forming the full concatenated matrix. The resulting approach achieves a favorable balance among speed, accuracy, and scalability, significantly enhancing the reliability and practicality of SVD compression in applications such as multi-view learning, signal processing, and neural network compression.

This work addresses the multi-level low-rank (MLR) matrix approximation problem under the Frobenius norm, tackling three core challenges: hierarchical structural partitioning (row/column stratification), rank allocation (optimizing individual block ranks under a total storage budget), and joint factor fitting. We propose the first end-to-end joint optimization framework for MLR matrices, unifying structural design, rank assignment, and factor learning within a single model. Our approach employs hierarchical block-diagonal parameterization, alternating optimization, and a constrained rank allocation algorithm to achieve coordinated optimization. The resulting approximation preserves matrix-vector multiplication complexity at O(n). Empirical evaluation on multiple benchmark datasets shows that our method reduces approximation error by 35% on average compared to single-level low-rank baselines, significantly improving both accuracy and storage efficiency. The implementation is publicly available.

This work addresses the lack of efficient solutions for computing a large batch of small-scale singular value decompositions (SVDs) on GPUs. The authors propose a GPU-accelerated batched SVD solver based on the one-sided Jacobi algorithm, co-designed with hardware architecture to exploit fine-grained parallelism, optimize memory access patterns, and support multiple floating-point precisions. Implemented on both NVIDIA and AMD GPU platforms, the solver demonstrates exceptional robustness and scalability across diverse matrix shapes, conditioning numbers, and precision configurations. Experimental results show that the proposed method significantly outperforms existing vendor-provided libraries and open-source solvers in terms of computational performance while maintaining numerical reliability.

This paper investigates the theoretical behavior of high-dimensional partial least squares (PLS) for dual-matrix data fusion, focusing on its ability to estimate and its fundamental limitations in recovering shared low-rank latent structure. Leveraging random matrix theory, we establish the first rigorous asymptotic characterization of PLS-SVD singular vectors’ alignment with true latent directions, quantitatively identifying the phase transition threshold—i.e., the critical signal-to-noise ratio and dimension ratio—governing successful versus failed latent reconstruction. Furthermore, we prove that, for detecting the common latent subspace, PLS-SVD is asymptotically superior to single-dataset PCA, with a theoretically guaranteed advantage. The analysis not only explains the counterintuitive failure of PLS in high dimensions but also precisely delineates its statistical limits and necessary conditions for validity as a multi-view dimensionality reduction method.

This work identifies and quantifies a previously overlooked issue in model merging—termed “spectral over-accumulation”—where knowledge sharing across tasks leads to redundant accumulation along dominant spectral directions, causing singular value inflation and biasing the merged model toward a shared subspace. To address this, the authors propose a training- and data-free post-processing method that leverages singular value decomposition to analyze the weight spectrum and calibrates singular values via a subspace overlap-aware rescaling strategy, thereby restoring spectral balance. Evaluated on vision and language benchmarks, the approach significantly enhances existing merging techniques, improving the accuracy of Task Arithmetic by 13.0% and achieving state-of-the-art performance.

This work proposes a general framework based on generalized singular value decomposition (GSVD) to represent modern neural networks as left-invertible, norm-preserving nonlinear mappings while preserving their input–output behavior. For the first time, GSVD theory is extended to generic neural network architectures, and a data-driven algorithm is introduced to estimate this representation from trained models, thereby aligning distances between the input space and the feature space. The resulting interpretable representation not only facilitates analyses of model invertibility and robustness but also demonstrates empirical effectiveness in adversarial perturbation detection. This approach establishes a theoretical foundation for diagnosing model bias and advancing invertible representation learning.

This work addresses the challenge of efficiently maintaining low-rank approximations of large-scale dynamic matrices—such as financial return panels—where conventional singular value decomposition (SVD) is computationally prohibitive and existing incremental methods struggle to balance accuracy, subspace stability, and efficiency. The authors propose a unified incremental SVD framework that supports row/column appending and rank-1 updates, introducing projection-based explicit update rules that reformulate Brand’s heuristic rank truncation into a quantifiable operation. For the first time, refresh scheduling is treated as a core design dimension, integrating principal angle tracking, error-ratio monitoring, and adaptive rank selection to systematically evaluate periodic and threshold-driven refresh strategies. Experiments on synthetic data streams and multi-asset ETF factor models demonstrate that, with appropriate rank and refresh frequency settings, the method achieves over 95% of full SVD accuracy at minimal computational overhead, making it suitable for high-frequency, batch-infeasible scenarios.