Random features and polynomial rules

2024-02-15

SciPost Physics

AI Summary

This work investigates the generalization performance of random feature models under high-dimensional Gaussian data, focusing on how prediction accuracy depends on the number of random features $N$, the sample size $P$, and the input dimensionality $D$. Departing from conventional proportional scaling assumptions, we establish a rigorous mapping between random feature models and equivalent polynomial kernels. For the first time, we derive universal generalization error curves in non-asymptotic, finite-dimensional regimes. Our theoretical framework integrates statistical mechanics, random matrix theory, and high-dimensional asymptotic analysis, augmented by polynomial expansions and kernel equivalence modeling. We show that when $N$ and $P$ scale as power laws in $D$, the average generalization error is precisely characterized. Theory and numerical experiments agree quantitatively across several orders of magnitude in $N$ and $P$, and remain accurate even for finite $D$ and nonzero ratios $P/D^K$ or $N/D^L$, uncovering a fundamental equivalence between random features and polynomial kernels.

Technology Category

• Machine Learning: Kernel Methods
• Reasoning under Uncertainty: Relational Probabilistic Models
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience

Abstract

Random features models play a distinguished role in the theory of deep learning, describing the behavior of neural networks close to their infinite-width limit. In this work, we present a thorough analysis of the generalization performance of random features models for generic supervised learning problems with Gaussian data. Our approach, built with tools from the statistical mechanics of disordered systems, maps the random features model to an equivalent polynomial model, and allows us to plot average generalization curves as functions of the two main control parameters of the problem: the number of random features NN and the size PP of the training set, both assumed to scale as powers in the input dimension DD. Our results extend the case of proportional scaling between NN, PP and DD. They are in accordance with rigorous bounds known for certain particular learning tasks and are in quantitative agreement with numerical experiments performed over many order of magnitudes of NN and PP. We find good agreement also far from the asymptotic limits where D o ∞D→∞ and at least one between P/D^KP/DK, N/D^LN/DL remains finite.

Problem

• Random Features Model
• Deep Learning
• Prediction Ability

Innovation

• Statistical Physics Methods
• Polynomial Model Transformation
• Theoretical-Experimental Consistency