🤖 AI Summary
This study investigates the theoretical mechanism by which the Gaussian RBF kernel converges to the linear kernel as the bandwidth parameter σ tends to infinity, and its implications for kernel PCA. Through asymptotic analysis in reproducing kernel Hilbert space (RKHS), the work establishes—for the first time—a quantitative relationship between the data geometric eccentricity ρ and the convergence rate of kernel methods. It is shown that, under large bandwidths, the RKHS embedding asymptotically aligns with the principal component frame of Euclidean space via a similarity transformation. Theoretically, Gaussian kernel PCA thus converges to classical linear PCA in this limit. Experiments confirm that the eccentricity ρ effectively predicts the convergence behavior of the leading principal directions across diverse datasets, highlighting the pivotal role of data geometry in determining the asymptotic performance of kernel methods.
📝 Abstract
We show that, up to isotropic scaling, the Gaussian RBF reproducing kernel Hilbert space (RKHS) is asymptotically isometric to Euclidean space in the large bandwidth limit. This strongly suggests that kernel-based constructions reliant on metric properties of the RKHS will yield results for Gaussian RBF kernels that similarly approach those of linear kernels for large bandwidths. The asymptotic behavior of Gaussian CKA can be understood in this light. We further consider kernel PCA, showing that Gaussian RBF eigenvalues, eigenprojections, and principal components all converge to those of classical (linear) PCA as bandwidth $σ\rightarrow \infty$. For a given data representation, both the RKHS feature embeddings and the orthogonal PCA eigenframes of the two kernel types differ asymptotically by a geometric similarity transformation, up to a residual of size $O \left (\fracρσ \right )^2$, where $ρ$ is a measure of geometric eccentricity of the representation, equal to the ratio of maximum to median pairwise distance between data examples. Experiments over a diverse collection of data sets demonstrate that $ρ$ provides a simple and reliable predictor of dataset-specific convergence behavior in the top principal directions.