Bayesian Kernel Regression for Functional Data

2025-03-17

AI Summary

This paper addresses functional output regression—a neglected yet critical supervised learning problem involving predictions of structured outputs such as spectra or probability density functions. We propose a novel Bayesian framework integrating full kernelization within a reproducing kernel Hilbert space (RKHS). Our key contribution is the first analytical derivation of the posterior predictive distribution over functional outputs in RKHS, enabled by a multitask-style functional covariance model that captures intrinsic correlations among output functions. The method retains model simplicity while efficiently handling high-dimensional nonlinearities. It supports principled uncertainty quantification and yields closed-form functional predictive distributions. Empirical evaluation on synthetic benchmarks and a materials science task—predicting electronic density of states—demonstrates substantial improvements in accuracy and generalization over baselines, confirming the approach’s robustness and practical utility.

Technology Category

• Machine Learning: Kernel Methods
• Reasoning under Uncertainty: Relational Probabilistic Models
• Intelligent Robots: State Estimation

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies

Abstract

In supervised learning, the output variable to be predicted is often represented as a function, such as a spectrum or probability distribution. Despite its importance, functional output regression remains relatively unexplored. In this study, we propose a novel functional output regression model based on kernel methods. Unlike conventional approaches that independently train regressors with scalar outputs for each measurement point of the output function, our method leverages the covariance structure within the function values, akin to multitask learning, leading to enhanced learning efficiency and improved prediction accuracy. Compared with existing nonlinear function-on-scalar models in statistical functional data analysis, our model effectively handles high-dimensional nonlinearity while maintaining a simple model structure. Furthermore, the fully kernel-based formulation allows the model to be expressed within the framework of reproducing kernel Hilbert space (RKHS), providing an analytic form for parameter estimation and a solid foundation for further theoretical analysis. The proposed model delivers a functional output predictive distribution derived analytically from a Bayesian perspective, enabling the quantification of uncertainty in the predicted function. We demonstrate the model's enhanced prediction performance through experiments on artificial datasets and density of states prediction tasks in materials science.

Problem

• Develops a functional output regression model using kernel methods.
• Handles high-dimensional nonlinearity with a simple model structure.
• Provides Bayesian predictive distribution for uncertainty quantification.

Innovation

• Bayesian kernel regression for functional data
• Leverages covariance structure for multitask learning
• Analytic parameter estimation in RKHS framework