Scalable Derivative Gaussian Processes via Exact Gradient Reduction

2026-06-01

AI Summary

In high-dimensional settings, Gaussian process inference with gradient observations suffers from prohibitive computational complexity of 𝒪(n³d³), hindering scalability. This work proposes TERA, which establishes—for the first time—that components of the gradient orthogonal to the line connecting a target point and its conditioning points are conditionally independent of the target function value. This insight enables dimension-agnostic exact conditional density reduction. By exploiting the gradient structure of stationary kernels, TERA compresses gradient information into at most m² directional derivatives and integrates them into local factors via Vecchia approximation. The method achieves optimal predictive accuracy while reducing per-target evaluation complexity to 𝒪(dm² + m⁶) time and 𝒪(dm² + m⁴) memory. Consequently, both computation time and peak GPU memory consumption remain nearly constant with respect to input dimension d, substantially outperforming existing approaches.

Abstract

Gradient observations can substantially improve Gaussian process (GP) surrogates, particularly in high-dimensional settings where function evaluations are expensive. However, exact inference with $n$ function values and $n$ full gradients in $d$ dimensions scales cubically in the joint state size, imposing an intractable $\mathcal{O}(n^3 d^3)$ computational bottleneck. We introduce TERA, a highly scalable derivative GP method based on target-specific exact gradient reduction. We prove that for stationary kernels, the gradient components orthogonal to the directions connecting the target and conditioning points are conditionally independent of the target function value; consequently, the exact conditional density is fully characterized by at most $m^2$ directional derivatives once a conditioning set of size $m$ is specified. By using these reduced, dimension-free conditionals as local factors in a Vecchia approximation, TERA effectively decouples $n$ and $d$ from the dense matrix inversion. This reduces the per-target evaluation cost to $\mathcal{O}(dm^2 + m^6)$ time and $\mathcal{O}(dm^2 + m^4)$ memory, leaving the underlying derivative GP model mathematically unchanged. Empirical evaluations demonstrate that TERA achieves state-of-the-art predictive accuracy while operating orders of magnitude faster than standard derivative GPs. Crucially, both computation time and peak GPU memory remain essentially flat with respect to $d$, enabling highly scalable inference in high-dimensional spaces.

Problem

• Gaussian processes
• gradient observations
• scalability
• high-dimensional inference
• computational complexity

Innovation

• Derivative Gaussian Processes
• Exact Gradient Reduction
• Vecchia Approximation
• Scalable Inference
• High-dimensional GP