Hyperparameter Optimization for Randomized Algorithms: A Case Study for Random Features

2024-06-30

arXiv.org

AI Summary

This work addresses the challenge in random feature regression (RFR) that sampling-distribution hyperparameters are non-differentiable and thus inaccessible to gradient-based optimization. We propose a gradient-free black-box optimization framework based on ensemble Kalman inversion (EKI), the first application of EKI to learning hyperparameter distributions for random features. The method circumvents assumptions of objective differentiability and sample-wise gradient availability, enabling high-dimensional, robust, and fully automated hyperparameter tuning. By integrating random feature mapping with Bayesian modeling principles, our framework significantly improves RFR’s prediction accuracy and stability across diverse tasks—including global sensitivity analysis, chaotic system integration, and Bayesian inverse problems in atmospheric dynamics—demonstrating both broad applicability and practical utility.

Technology Category

• Machine Learning: Auto ML and Hyperparameter Tuning
• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Metareasoning and Metaheuristics

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Abstract

Randomized algorithms exploit stochasticity to reduce computational complexity. One important example is random feature regression (RFR) that accelerates Gaussian process regression (GPR). RFR approximates an unknown function with a random neural network whose hidden weights and biases are sampled from a probability distribution. Only the final output layer is fit to data. In randomized algorithms like RFR, the hyperparameters that characterize the sampling distribution greatly impact performance, yet are not directly accessible from samples. This makes optimization of hyperparameters via standard (gradient-based) optimization tools inapplicable. Inspired by Bayesian ideas from GPR, this paper introduces a random objective function that is tailored for hyperparameter tuning of vector-valued random features. The objective is minimized with ensemble Kalman inversion (EKI). EKI is a gradient-free particle-based optimizer that is scalable to high-dimensions and robust to randomness in objective functions. A numerical study showcases the new black-box methodology to learn hyperparameter distributions in several problems that are sensitive to the hyperparameter selection: two global sensitivity analyses, integrating a chaotic dynamical system, and solving a Bayesian inverse problem from atmospheric dynamics. The success of the proposed EKI-based algorithm for RFR suggests its potential for automated optimization of hyperparameters arising in other randomized algorithms.

Problem

• Optimizes hyperparameters for randomized algorithms
• Introduces gradient-free ensemble Kalman inversion
• Enhances random feature regression performance

Innovation

• Ensemble Kalman inversion optimization
• Vector-valued random features tuning
• Gradient-free hyperparameter learning