Convergence for adaptive resampling of random Fourier features

2025-09-03

AI Summary

To address the limited representational capacity of Random Fourier Features (RFF) in high-dimensional data—caused by uniform spectral sampling—this paper proposes a data-adaptive, asymptotically optimal frequency resampling method. The approach integrates an adaptive random walk strategy to dynamically refine the Fourier frequency distribution during training and provides theoretical guarantees of convergence as both the number of features and sample size tend to infinity. To enhance computational efficiency, the resampling procedure is embedded within a conjugate gradient iteration framework for approximate least-squares solution. Experiments demonstrate that the proposed method significantly outperforms standard RFF and existing adaptive variants on both regression and classification tasks. Crucially, it retains linear time complexity while achieving superior prediction accuracy and scalability, making it well-suited for large-scale, high-dimensional applications.

Technology Category

• Machine Learning: Kernel Methods
• Search and Optimization: Sampling/Simulation-based Search
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: User modeling for targeted and personalized online advertising

Abstract

The machine learning random Fourier feature method for data in high dimension is computationally and theoretically attractive since the optimization is based on a convex standard least squares problem and independent sampling of Fourier frequencies. The challenge is to sample the Fourier frequencies well. This work proves convergence of a data adaptive method based on resampling the frequencies asymptotically optimally, as the number of nodes and amount of data tend to infinity. Numerical results based on resampling and adaptive random walk steps together with approximations of the least squares problem by conjugate gradient iterations confirm the analysis for regression and classification problems.

Problem

• Optimizing Fourier frequency sampling in high-dimensional data
• Proving convergence for adaptive resampling methods asymptotically
• Validating approach with numerical regression and classification results

Innovation

• Adaptive resampling of Fourier frequencies
• Conjugate gradient for least squares
• Random walk steps for optimization