Iterative Methods for Full-Scale Gaussian Process Approximations for Large Spatial Data

2024-05-23

arXiv.org

AI Summary

Gaussian processes (GPs) face prohibitive computational complexity and poor scalability when modeling large-scale spatial data. To address this, we propose the Full-Scale Approximation (FSA) framework, which unifies global and local covariance structures within a single coherent model. We introduce a novel adaptive preconditioner that substantially accelerates convergence and enhances robustness of the conjugate gradient method for solving linear systems arising from GP inference. Additionally, we design an efficient prediction variance estimator based on stochastic trace approximation, circumventing costly matrix inversions. Empirical evaluations on both synthetic and real-world datasets demonstrate that FSA achieves accuracy comparable to exact inference via Cholesky decomposition while reducing computation time by orders of magnitude. The core implementation is provided as an open-source C++ library, with Python and R interfaces for broad accessibility.

Technology Category

• Machine Learning: Kernel Methods
• Reasoning under Uncertainty: Probabilistic Programming
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

Gaussian processes are flexible probabilistic regression models which are widely used in statistics and machine learning. However, a drawback is their limited scalability to large data sets. To alleviate this, we consider full-scale approximations (FSAs) that combine predictive process methods and covariance tapering, thus approximating both global and local structures. We show how iterative methods can be used to reduce the computational costs for calculating likelihoods, gradients, and predictive distributions with FSAs. We introduce a novel preconditioner and show that it accelerates the conjugate gradient method's convergence speed and mitigates its sensitivity with respect to the FSA parameters and the eigenvalue structure of the original covariance matrix, and we demonstrate empirically that it outperforms a state-of-the-art pivoted Cholesky preconditioner. Further, we present a novel, accurate, and fast way to calculate predictive variances relying on stochastic estimations and iterative methods. In both simulated and real-world data experiments, we find that our proposed methodology achieves the same accuracy as Cholesky-based computations with a substantial reduction in computational time. Finally, we also compare different approaches for determining inducing points in predictive process and FSA models. All methods are implemented in a free C++ software library with high-level Python and R packages.

Problem

• Reducing computational costs in Gaussian process approximations
• Accelerating convergence speed for large spatial data
• Improving predictive variance calculations with iterative methods

Innovation

• Combines predictive process and covariance tapering
• Introduces novel preconditioner for conjugate gradient
• Uses stochastic simulation for predictive variances