Adaptive RKHS Fourier Features for Compositional Gaussian Process Models

2024-07-01

arXiv.org

AI Summary

This work addresses the limited capability of compositional Gaussian processes (CGPs) in modeling nonstationary processes. We propose an ODE-driven adaptive RKHS Fourier feature method, which incorporates ordinary differential equation constraints into Fourier basis learning to enable dynamic modulation of amplitude and phase. These learned features are integrated into a linear-transformation-based CGP architecture via convolutional parameterization, and trained efficiently using doubly stochastic variational inference. Unlike conventional local inducing-point approximations, our approach overcomes their global modeling limitations and substantially improves capture of complex nonstationary patterns. Experiments demonstrate that the proposed model achieves significantly higher predictive accuracy than state-of-the-art CGPs and nonstationary GPs across multiple regression benchmarks. This work establishes a novel paradigm for deep generative modeling of nonstationary processes.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Search and Optimization: Non-convex Optimization
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications

Abstract

Deep Gaussian Processes (DGPs) leverage a compositional structure to model non-stationary processes. DGPs typically rely on local inducing point approximations across intermediate GP layers. Recent advances in DGP inference have shown that incorporating global Fourier features from the Reproducing Kernel Hilbert Space (RKHS) can enhance the DGPs' capability to capture complex non-stationary patterns. This paper extends the use of these features to compositional GPs involving linear transformations. In particular, we introduce Ordinary Differential Equation(ODE)--based RKHS Fourier features that allow for adaptive amplitude and phase modulation through convolution operations. This convolutional formulation relates our work to recently proposed deep latent force models, a multi-layer structure designed for modelling nonlinear dynamical systems. By embedding these adjustable RKHS Fourier features within a doubly stochastic variational inference framework, our model exhibits improved predictive performance across various regression tasks.

Problem

• Extends RKHS Fourier features to compositional Gaussian Processes
• Introduces adaptive ODE-based RKHS features for modulation
• Enhances predictive performance in regression tasks

Innovation

• Adaptive RKHS Fourier features enhance DGPs
• ODE-based features allow amplitude-phase modulation
• Doubly stochastic variational inference improves performance