🤖 AI Summary
This work addresses the lack of theoretical guarantees for nonparametric regression in reproducing kernel Hilbert spaces under model misspecification, high-dimensional settings, and nonconvex losses. It establishes a unified theoretical framework for regularized M-estimators encompassing a broad class of both convex and nonconvex loss functions. By introducing a novel complexity measure, the analysis achieves an explicit bias–variance decomposition. Leveraging tools from functional analysis and empirical process theory, the study proves the existence, measurability, and asymptotic linearity of the estimator without requiring closed-form solutions or global Lipschitz assumptions. Notably, within tensor-product Sobolev spaces, the framework reveals a mechanism to circumvent the curse of dimensionality, yielding minimax-optimal convergence rates that depend on mixed smoothness of the underlying function. The variance component is shown to be robust to model misspecification, and numerical experiments in C++ corroborate the theoretical findings.
📝 Abstract
We develop a comprehensive theory for regularized M-estimation in reproducing kernel Hilbert spaces. Under mild conditions on the loss we establish existence and measurability of the estimator, covering a wide range of convex and non-convex losses, including bounded robust losses. We further prove sharp rates of convergence with an explicit bias-variance decomposition governed by a novel complexity measure. We show that the variance is independent of misspecification, while the bias depends on a source condition parameter known in the learning literature. For tensor product Sobolev spaces we obtain new rates that connect to spaces of functions with dominating mixed smoothness, substantially extending existing results and explaining why these estimators circumvent the curse of dimensionality. Our methodology, combining elements from both functional analysis and empirical process theory, allows for an asymptotic linearisation of the objective function that avoids both closed-form solutions and global Lipschitz assumptions, and may be of independent interest. The estimators are implemented in C++ and theory is supported by numerical experiments.