On the kernel learning problem

2025-02-17

AI Summary

This paper addresses the low modeling efficiency of kernel ridge regression (KRR) on multiscale data. We propose a generalized KRR framework incorporating a learnable structured matrix parameter $U$, casting kernel selection as a joint nonlinear variational optimization problem with $U$. This enables automatic identification of salient feature variables and intrinsic scale parameters. Leveraging reproducing kernel Hilbert space theory and multiscale analysis, we rigorously establish existence, stability, and multiscale response properties of the $U$-optimized solution. Compared to standard KRR, our method significantly improves fitting accuracy and generalization performance on heterogeneous-scale data. The framework provides a theoretically grounded yet practical paradigm for data-driven kernel learning.

Technology Category

Machine Learning: Kernel MethodsComputer Vision: Learning & Optimization for CVSearch and Optimization: Learning to Search

Application Category

Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for rankingGraph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphsSemantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies

Abstract

The classical kernel ridge regression problem aims to find the best fit for the output $Y$ as a function of the input data $Xin mathbb{R}^d$, with a fixed choice of regularization term imposed by a given choice of a reproducing kernel Hilbert space, such as a Sobolev space. Here we consider a generalization of the kernel ridge regression problem, by introducing an extra matrix parameter $U$, which aims to detect the scale parameters and the feature variables in the data, and thereby improve the efficiency of kernel ridge regression. This naturally leads to a nonlinear variational problem to optimize the choice of $U$. We study various foundational mathematical aspects of this variational problem, and in particular how this behaves in the presence of multiscale structures in the data.

Problem

• Optimize kernel ridge regression
• Introduce matrix parameter for scale detection
• Analyze variational problem with multiscale data

Innovation

• Generalized kernel ridge regression
• Matrix parameter for scale detection
• Nonlinear variational optimization