A reproducible comparative study of categorical kernels for Gaussian process regression, with new clustering-based nested kernels

2025-10-02

AI Summary

Designing kernels for Gaussian process regression with mixed continuous and categorical inputs has long suffered from inconsistent evaluation criteria, non-reproducible implementations, and limited generalization. This paper establishes the first reproducible, systematic benchmarking framework to comprehensively evaluate existing categorical kernels. We propose a novel nested kernel approach that jointly leverages prior-free clustering and target encoding to automatically uncover latent group structures in categorical variables. Crucially, we introduce an optimization-aware evaluation metric that prioritizes predictive performance under computational constraints. Our method achieves significantly improved generalization while maintaining low computational overhead. Empirical results demonstrate that the nested kernel substantially outperforms state-of-the-art baselines on datasets exhibiting implicit group structure. Moreover, even when such structure is unknown a priori, our clustering strategy attains optimal prediction accuracy at lower algorithmic complexity than competing methods.

Technology Category

• Machine Learning: Kernel Methods
• Search and Optimization: Mixed Discrete/Continuous Search
• Constraint Satisfaction and Optimization: Mixed Discrete/Continuous Optimization

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Designing categorical kernels is a major challenge for Gaussian process regression with continuous and categorical inputs. Despite previous studies, it is difficult to identify a preferred method, either because the evaluation metrics, the optimization procedure, or the datasets change depending on the study. In particular, reproducible code is rarely available. The aim of this paper is to provide a reproducible comparative study of all existing categorical kernels on many of the test cases investigated so far. We also propose new evaluation metrics inspired by the optimization community, which provide quantitative rankings of the methods across several tasks. From our results on datasets which exhibit a group structure on the levels of categorical inputs, it appears that nested kernels methods clearly outperform all competitors. When the group structure is unknown or when there is no prior knowledge of such a structure, we propose a new clustering-based strategy using target encodings of categorical variables. We show that on a large panel of datasets, which do not necessarily have a known group structure, this estimation strategy still outperforms other approaches while maintaining low computational cost.

Problem

• Comparing categorical kernels for Gaussian process regression with mixed inputs
• Addressing reproducibility issues in previous categorical kernel evaluations
• Proposing clustering-based kernels when group structure is unknown

Innovation

• Developed new clustering-based nested kernels
• Proposed target encoding strategy for unknown structures
• Introduced quantitative metrics for reproducible kernel evaluation