STRIDE: Sparse Techniques for Regression in Deep Gaussian Processes

2025-05-16

AI Summary

Gaussian processes (GPs) and their deep extensions face significant challenges in large-scale and multi-scale function modeling, including low training efficiency, poor uncertainty calibration, and difficulty in jointly optimizing inducing point locations and deep kernel hyperparameters. To address these issues, we propose a particle-based Expectation-Maximization (EM) framework that, for the first time, embeds particle filtering into a variational EM procedure for deep sparse GPs. Our method integrates MCMC sampling with variational inference, enabling joint optimization of inducing points and deep kernel parameters while preserving computational scalability. Crucially, it substantially improves posterior uncertainty characterization without sacrificing efficiency. On standard benchmarks, our approach achieves superior predictive accuracy and more reliable uncertainty quantification compared to state-of-the-art deep GP methods, while also demonstrating significantly faster training convergence.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Relational Probabilistic Models
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Gaussian processes (GPs) have gained popularity as flexible machine learning models for regression and function approximation with an in-built method for uncertainty quantification. However, GPs suffer when the amount of training data is large or when the underlying function contains multi-scale features that are difficult to represent by a stationary kernel. To address the former, training of GPs with large-scale data is often performed through inducing point approximations (also known as sparse GP regression (GPR)), where the size of the covariance matrices in GPR is reduced considerably through a greedy search on the data set. To aid the latter, deep GPs have gained traction as hierarchical models that resolve multi-scale features by combining multiple GPs. Posterior inference in deep GPs requires a sampling or, more usual, a variational approximation. Variational approximations lead to large-scale stochastic, non-convex optimisation problems and the resulting approximation tends to represent uncertainty incorrectly. In this work, we combine variational learning with MCMC to develop a particle-based expectation-maximisation method to simultaneously find inducing points within the large-scale data (variationally) and accurately train the GPs (sampling-based). The result is a highly efficient and accurate methodology for deep GP training on large-scale data. We test our method on standard benchmark problems.

Problem

• Addressing scalability issues in Gaussian processes for large datasets
• Improving multi-scale feature representation in deep Gaussian processes
• Combining variational learning with MCMC for accurate uncertainty quantification

Innovation

• Combines variational learning with MCMC
• Uses particle-based expectation-maximisation method
• Efficient deep GP training on large-scale data