Have ASkotch: A Neat Solution for Large-scale Kernel Ridge Regression

2024-07-14

AI Summary

Large-scale kernel ridge regression (KRR) suffers from prohibitive computational and memory complexity, hindering its scalability to big-data regimes; while existing inducing-point approximations improve scalability, they incur substantial prediction accuracy loss. This paper introduces ASkotch—a novel, exact KRR solver achieving the first linear convergence guarantee independent of the condition number. Its core innovations integrate randomized preconditioning, accelerated gradient iteration, adaptive sampling via ridge leverage scores, and determinant point process theory—yielding an exact, bias-free, and provably convergent KRR framework. Evaluated on 23 large-scale regression and classification benchmarks across diverse domains, ASkotch consistently outperforms state-of-the-art methods in both accuracy and efficiency. It enables practical deployment of exact KRR in high-stakes applications such as computational chemistry and clinical prediction.

Technology Category

• Machine Learning: Kernel Methods
• Search and Optimization: Learning to Search
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Security and Privacy: Large-scale security measurements

Abstract

Kernel ridge regression (KRR) is a fundamental computational tool, appearing in problems that range from computational chemistry to health analytics, with a particular interest due to its starring role in Gaussian process regression. However, full KRR solvers are challenging to scale to large datasets: both direct (i.e., Cholesky decomposition) and iterative methods (i.e., PCG) incur prohibitive computational and storage costs. The standard approach to scale KRR to large datasets chooses a set of inducing points and solves an approximate version of the problem, inducing points KRR. However, the resulting solution tends to have worse predictive performance than the full KRR solution. In this work, we introduce a new solver, ASkotch, for full KRR that provides better solutions faster than state-of-the-art solvers for full and inducing points KRR. ASkotch is a scalable, accelerated, iterative method for full KRR that provably obtains linear convergence. Under appropriate conditions, we show that ASkotch obtains condition-number-free linear convergence. This convergence analysis rests on the theory of ridge leverage scores and determinantal point processes. ASkotch outperforms state-of-the-art KRR solvers on a testbed of 23 large-scale KRR regression and classification tasks derived from a wide range of application domains, demonstrating the superiority of full KRR over inducing points KRR. Our work opens up the possibility of as-yet-unimagined applications of full KRR across a number of disciplines.

Problem

• Scalable solution for full kernel ridge regression
• Improved predictive performance over inducing points KRR
• Achieves linear convergence in large-scale datasets

Innovation

• Scalable full KRR solver
• Linear convergence guarantee
• Outperforms state-of-the-art methods