Worth Their Weight: Randomized and Regularized Block Kaczmarz Algorithms without Preprocessing

2025-02-02

AI Summary

For large-scale linear least-squares problems, existing Randomized Block Kaczmarz (RBK) methods suffer from convergence instability, variance explosion, and condition-number sensitivity—particularly under rapidly decaying singular value spectra—due to reliance on complex sampling schemes or preprocessing. This paper proposes ReBlocK: the first method to theoretically establish convergence of uniformly sampled block Kaczmarz iterations to a weighted least-squares solution; it incorporates Tikhonov regularization to stabilize the iterative process; and it designs an efficient update rule by integrating randomized block selection with natural gradient principles. Theoretical analysis is grounded in a Monte Carlo convergence framework that rigorously decouples variance from condition-number dependence. Experiments demonstrate that ReBlocK significantly outperforms mini-batch SGD on problems with fast-decaying spectra, achieving both strong theoretical guarantees and practical efficiency.

Technology Category

• Reasoning under Uncertainty: Stochastic Optimization
• Machine Learning: Optimization
• Search and Optimization: Non-convex Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Security and Privacy: Large-scale security measurements

Abstract

Due to the ever growing amounts of data leveraged for machine learning and scientific computing, it is increasingly important to develop algorithms that sample only a small portion of the data at a time. In the case of linear least-squares, the randomized block Kaczmarz method (RBK) is an appealing example of such an algorithm, but its convergence is only understood under sampling distributions that require potentially prohibitively expensive preprocessing steps. To address this limitation, we analyze RBK when the data is sampled uniformly, showing that its iterates converge in a Monte Carlo sense to a $ extit{weighted}$ least-squares solution. Unfortunately, for general problems the condition number of the weight matrix and the variance of the iterates can become arbitrarily large. We resolve these issues by incorporating regularization into the RBK iterations. Numerical experiments, including examples arising from natural gradient optimization, suggest that the regularized algorithm, ReBlocK, outperforms minibatch stochastic gradient descent for realistic problems that exhibit fast singular value decay.

Problem

• Linear Least Squares
• Randomized Block Kaczmarz Method
• Singular Value Decay

Innovation

• ReBlocK
• Regularization
• Fast Singular Value Decay