Fine-grained Analysis and Faster Algorithms for Iteratively Solving Linear Systems

2024-05-09

arXiv.org

AI Summary

Existing iterative solvers for large-scale linear systems suffer from strong dependence on the global condition number and coarse-grained complexity analyses. Method: We introduce the *spectral tail condition number* $kappa_ell$, a new fine-grained spectral measure, and develop a refined time-complexity framework. Our approach formally defines $kappa_ell$, integrates it with the Sketch-and-Project paradigm, Nesterov acceleration, determinant point process sampling, and universality theory for Gaussian matrices, thereby exposing an intrinsic connection between iteration complexity and the matrix multiplication exponent $omega$. Contribution/Results: Our analysis achieves a sharper separation between deterministic and randomized algorithms, yielding an $ ilde{O}(kappa_ell n^2 log(1/varepsilon))$ bound for computing an $varepsilon$-accurate solution—valid for $ell$ up to $O(n^{0.729})$. This significantly improves the fine-grained analysis of the conjugate gradient method and establishes a novel theoretical benchmark for iterative algorithm design.

Technology Category

• Machine Learning: Matrix & Tensor Methods
• Search and Optimization: Non-convex Optimization
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Security and Privacy: Large-scale security measurements
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

While effective in practice, iterative methods for solving large systems of linear equations can be significantly affected by problem-dependent condition number quantities. This makes characterizing their time complexity challenging, particularly when we wish to make comparisons between deterministic and stochastic methods, that may or may not rely on preconditioning and/or fast matrix multiplication. In this work, we consider a fine-grained notion of complexity for iterative linear solvers which we call the spectral tail condition number, $kappa_ell$, defined as the ratio between the $ell$th largest and the smallest singular value of the matrix representing the system. Concretely, we prove the following main algorithmic result: Given an $n imes n$ matrix $A$ and a vector $b$, we can find $ ilde{x}$ such that $|A ilde{x}-b|leqepsilon|b|$ in time $ ilde{O}(kappa_ellcdot n^2log 1/epsilon)$ for any $ell = O(n^{frac1{omega-1}})=O(n^{0.729})$, where $omega approx 2.372$ is the current fast matrix multiplication exponent. This guarantee is achieved by Sketch-and-Project with Nesterov's acceleration. Some of the implications of our result, and of the use of $kappa_ell$, include direct improvement over a fine-grained analysis of the Conjugate Gradient method, suggesting a stronger separation between deterministic and stochastic iterative solvers; and relating the complexity of iterative solvers to the ongoing algorithmic advances in fast matrix multiplication, since the bound on $ell$ improves with $omega$. Our main technical contributions are new sharp characterizations for the first and second moments of the random projection matrix that commonly arises in sketching algorithms, building on a combination of techniques from combinatorial sampling via determinantal point processes and Gaussian universality results from random matrix theory.

Problem

• Quantify cost of solving large linear systems efficiently
• Improve stochastic algorithms for low-dimensional structured data
• Analyze random projection matrices in Sketch-and-Project methods

Innovation

• Fine-grained complexity analysis for linear systems
• Stochastic algorithm using Sketch-and-Project paradigm
• Improved runtime via random projection matrix analysis