Weighted Euclidean Distance Matrices over Mixed Continuous and Categorical Inputs for Gaussian Process Models

2025-03-04

AI Summary

Standard Gaussian processes (GPs) struggle to model mixed inputs containing categorical variables due to the absence of a suitable correlation structure for categorical domains. To address this, we propose the Weighted Euclidean Gaussian Process (WEGP), the first GP framework that defines categorical distances as learnable linear combinations of base distance matrices and embeds this formulation within a fully Bayesian inference framework—enabling joint end-to-end inference of distance weights and kernel hyperparameters. We theoretically establish that the resulting kernel is guaranteed to be positive definite and satisfies performance bounds. WEGP integrates three key innovations: decomposition of base distance matrices, Bayesian optimization-aware ensemble design, and a customized hybrid kernel architecture. Extensive experiments on synthetic benchmarks and real-world optimization tasks demonstrate that WEGP achieves significantly improved predictive accuracy and Bayesian optimization efficiency, consistently outperforming state-of-the-art GP methods for mixed inputs.

Technology Category

• Search and Optimization: Mixed Discrete/Continuous Search
• Machine Learning: Bayesian Learning
• Constraint Satisfaction and Optimization: Mixed Discrete/Continuous Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Gaussian Process (GP) models are widely utilized as surrogate models in scientific and engineering fields. However, standard GP models are limited to continuous variables due to the difficulties in establishing correlation structures for categorical variables. To overcome this limitati on, we introduce WEighted Euclidean distance matrices Gaussian Process (WEGP). WEGP constructs the kernel function for each categorical input by estimating the Euclidean distance matrix (EDM) among all categorical choices of this input. The EDM is represented as a linear combination of several predefined base EDMs, each scaled by a positive weight. The weights, along with other kernel hyperparameters, are inferred using a fully Bayesian framework. We analyze the predictive performance of WEGP theoretically. Numerical experiments validate the accuracy of our GP model, and by WEGP, into Bayesian Optimization (BO), we achieve superior performance on both synthetic and real-world optimization problems.

Problem

• Overcome GP model limitation with mixed continuous and categorical inputs
• Introduce WEGP for constructing kernel functions for categorical variables
• Validate WEGP accuracy and superior performance in Bayesian Optimization

Innovation

• WEGP integrates categorical inputs via Euclidean distance matrices.
• Bayesian framework infers weights for kernel hyperparameters.
• WEGP enhances Bayesian Optimization with superior performance.