polar decomposition

Low-rank adaptation (LoRA) of large language models suffers from low subspace utilization and suboptimal fine-tuning performance due to the significant gap between stable rank and algebraic rank in learned low-rank updates. Method: This work introduces polar decomposition into LoRA parameterization for the first time, factorizing the low-rank update into two orthogonal matrices and a diagonal scaling matrix, with constraints imposed on the Stiefel manifold. Contribution/Results: We theoretically prove that this design enables exponential acceleration in convergence rate. Coupled with Riemannian optimization, it substantially improves subspace utilization efficiency. Empirical evaluation across three major benchmarks—language understanding, commonsense reasoning, and mathematical problem solving—demonstrates consistent and significant performance gains on base models ranging from 350M to 27B parameters, validating both generalizability and effectiveness.

This work addresses low-rank optimization problems—including orthogonal-constrained quadratic optimization and matrix completion—by introducing a novel relaxation-sampling paradigm. Methodologically, it pioneers a semidefinite relaxation framework inspired by the Goemans–Williamson approach, yielding a tighter and more general relaxation formulation; leveraging block-wise matrix sparsity, it substantially reduces the number of optimization variables, and integrates randomized rounding for efficient solution recovery. Theoretically, the method provides provable approximation guarantees for near-optimality. Empirically, it achieves superior accuracy and scalability: problem sizes solvable are improved by several orders of magnitude, significantly expanding the practical applicability of low-rank modeling to large-scale orthogonal optimization and matrix completion tasks.

To address the high computational cost of SVD approximation updates in streaming data and the scalability limitations of existing incremental/truncated SVD methods at large truncation ranks, this paper proposes a low-rank update framework based on Bidirectional Diagonal Decomposition (BDD). Our method enables efficient rank-$r$ updates via three key innovations: (1) a compact Householder transformation reducing memory usage by 50%; (2) Givens rotations enabling $O(r^2)$-complexity rank-$r$ updates; and (3) a hybrid sparse-plus-low-rank separation strategy for accurate and scalable matrix approximation. Experiments on recommendation systems and network subspace tracking demonstrate that our approach significantly outperforms LAPACK SVD and state-of-the-art incremental SVD methods—achieving superior accuracy, real-time performance even at high truncation ranks, and balanced throughput–precision trade-offs.

To address the slow convergence of the classical multiplicative update (MU) algorithm for nonnegative matrix factorization (NMF), this paper proposes the fast multiplicative update (fastMU) algorithm. Methodologically, fastMU reformulates MU as an alternating majorization-minimization (MM) procedure—its first such interpretation—and constructs tighter upper bounds on the Hessian for each subproblem, thereby accelerating optimization under both the Frobenius norm (quadratic loss) and the generalized β-divergence. Theoretical analysis guarantees monotonic objective descent and nonnegativity preservation throughout iterations. Empirically, fastMU achieves speedups of several orders of magnitude over standard MU on both synthetic and real-world datasets, while matching the state-of-the-art performance under Frobenius loss. This work thus unifies theoretical rigor with practical efficiency in NMF optimization.

To address the constraint in federated learning that clients’ private data cannot be shared, this paper proposes a data-free distributed low-rank matrix factorization method. The core method introduces power initialization into federated matrix factorization for the first time, transforming the original non-convex problem into a sequence of smooth, strongly convex subproblems, and designs a parallel Nesterov-accelerated federated optimization algorithm requiring only one global communication round during initialization. Theoretically, the method is proven to achieve linear convergence—significantly outperforming existing distributed approaches—and provides a tight upper bound on the Frobenius norm of the reconstruction error. Extensive experiments on both synthetic and real-world datasets validate its efficiency, robustness, and superior communication efficiency.

This work addresses the limitations of traditional interior-point methods in non-negative matrix factorization (NMF), which often suffer from slow convergence or suboptimal solutions due to enforcing non-negativity constraints throughout optimization. The authors propose eNMF, an exterior-point framework that decouples low-rank approximation from non-negativity constraints for the first time: it first computes an unconstrained optimal factorization and then maps the factors near the non-negative orthant boundary via factor rotation, followed by an iterative update strategy satisfying the Karush–Kuhn–Tucker (KKT) conditions. This approach offers a geometric perspective on NMF, revealing equivalence classes of factors under permutation and orthogonal transformations and explaining why diverse algorithms often converge to equivalent solutions. Experiments show that eNMF outperforms 81 baseline variants across five datasets, achieving up to 30% lower reconstruction error at equal runtime and up to 150% faster convergence at comparable accuracy, while significantly improving downstream performance in audio and recommendation tasks.

This work addresses the challenge of lacking a general computable convex relaxation for quadratic matrix optimization problems with rank constraints by proposing a lifted semidefinite relaxation framework that does not rely on spectral structure assumptions. By analyzing block redundancies in moment matrices, the authors derive a compact equivalent formulation involving only two small-scale semidefinite constraints. They further design projection-based cutting planes that exploit rank inheritance under linear mappings to strengthen the low-rank constraint. This approach significantly enhances scalability and enables efficient solutions for problems such as matrix completion and reduced-rank regression, handling instances with dimensions up to \(n + m\).

This work addresses the longstanding challenge in operator learning of simultaneously achieving stability, interpretability, and efficient parallelization, which stems from the lack of explicit spectral structure modeling. The authors propose a polar-spectral operator framework that leverages polar-coordinate geometry to map problems into the spectral domain, where they are decomposed into orthogonal eigenmodes processed independently. A self-adjoint-inspired spectral constraint mechanism is introduced, which not only reduces parameter count and computational complexity but also naturally yields a novel mode-based model parallelization strategy. Experiments demonstrate that the method enables stable training on MNIST, significantly improves convergence, and produces more interpretable and computationally efficient model representations.

This work proposes an averaged nonmonotone alternating update method (A-NAUM) to address the pervasive challenge of generalized symmetric matrix factorization, which is typically nonconvex, nonsmooth, and non-Lipschitz in machine learning and imaging sciences. By introducing auxiliary variable splitting and a symmetry-induced exact penalty term, the authors construct an equivalent relaxed merit function and, for the first time, integrate an averaged nonmonotone line search with Kurdyka–Łojasiewicz (KL) convergence analysis. Theoretically, the method guarantees global convergence of the iterative sequence to a critical point and provides explicit convergence rates. Numerical experiments on real-world datasets demonstrate its efficiency and robustness.

This work addresses the inefficiency of symmetric nonnegative matrix factorization (Symmetric NMF) in graph clustering and related tasks, which stems from the slow convergence of conventional projected gradient methods. To overcome this limitation, the authors propose SNMPBB, the first algorithm to integrate a nonmonotone projected Barzilai–Borwein method into Symmetric NMF, achieving both rapid and globally convergent optimization while preserving curvature information for enhanced efficiency. The framework is further extended to Graph-SNMPBB, incorporating graph Laplacian regularization for improved clustering performance, and LAI-SNMPBB, which combines low-rank approximation with randomization for scalability to large sparse matrices. Experiments demonstrate that SNMPBB is six times faster than SymANLS on synthetic data; Graph-SNMPBB matches or exceeds state-of-the-art accuracy on six real-world clustering benchmarks; and LAI-SNMPBB significantly outperforms LAI-SymPGNCG across 34 SuiteSparse matrices.