Fast Multitask Gaussian Process Regression

2026-03-16

AI Summary

This work addresses the scalability limitations of multi-task Gaussian process regression under large sample sizes, which arise from high storage and computational complexity. The authors propose a fast multi-task Gaussian process method that introduces low-discrepancy design points for each task and employs a specialized product kernel to induce structured block Gram matrices—such as block circulant structures—in the covariance. Efficient algorithms are developed for storage, matrix inversion, and determinant computation leveraging this structure. This approach constitutes the first extension of fast single-task Gaussian processes to the multi-task setting, accommodating heterogeneous sampling nodes and varying sample sizes across tasks, and further enables a fast Bayesian numerical integration scheme. Experiments demonstrate substantial computational gains over standard methods when the number of tasks is modest but sample sizes are large. The implementation is open-sourced and integrated into the FastGPs package.

Technology Category

• Machine Learning: Kernel Methods
• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

Gaussian process (GP) regression is a powerful probabilistic modeling technique with built-in uncertainty quantification. When one has access to multiple correlated simulations (tasks), it is common to fit a multitask GP (MTGP) surrogate which is capable of capturing both inter-task and intra-task correlations. However, with a total of $N$ evaluations across all tasks, fitting an MTGP is often infeasible due to the $\mathcal{O}(N^2)$ storage and $\mathcal{O}(N^3)$ computations required to store, solve a linear system in, and compute the determinant of the $N \times N$ Gram matrix of pairwise kernel evaluations. In the single-task setting, one may reduce the required storage to $\mathcal{O}(N)$ and computations to $\mathcal{O}(N \log N)$ by fitting "fast GPs" which pair low-discrepancy design points from quasi-Monte Carlo to special kernel forms which yields nicely structured Gram matrices, e.g., circulant matrices. This article generalizes fast GPs to fast MTGPs which pair low-discrepancy design points for each task to special product kernel forms which yields nicely structured block Gram matrices, e.g., circulant block matrices. An algorithm is presented to efficiently store, invert, and compute the determinant of such Gram matrices with optionally different sampling nodes and different sample sizes for each task. Derivations for fast MTGP Bayesian cubature are also provided. A GPU-compatible, open-source Python implementation is made available in the FastGPs package (https://alegresor.github.io/fastgps/). We validate the efficiency of our algorithm and implementation compared to standard techniques on a range of problems with low numbers of tasks and large sample sizes.

Problem

• multitask Gaussian process
• computational complexity
• Gram matrix
• scalability
• Bayesian modeling

Innovation

• multitask Gaussian process
• fast GP
• structured Gram matrix
• low-discrepancy sequences
• Bayesian cubature