Gradient-based Sample Selection for Faster Bayesian Optimization

2025-04-10

AI Summary

Bayesian optimization (BO) suffers from poor scalability to large-budget settings due to the $O(n^3)$ time complexity of Gaussian process (GP) modeling. This work proposes a gradient-driven subset selection mechanism that significantly reduces GP fitting cost while preserving surrogate model fidelity. For the first time, gradient information is leveraged to jointly assess sample diversity and representativeness, enabling theoretically grounded sublinear regret—specifically, $O(sqrt{T}log T)$—thereby breaking the computational bottleneck inherent in full-data GP modeling. The method integrates GP regression, gradient-guided sampling, and subset optimization. Empirical evaluation on synthetic and real-world benchmarks demonstrates several-fold speedup in GP fitting time, while maintaining optimization performance comparable to standard BO. Thus, the approach achieves a favorable trade-off between computational efficiency and optimization effectiveness.

Technology Category

• Search and Optimization: Sampling/Simulation-based Search
• Machine Learning: Optimization
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

Bayesian optimization (BO) is an effective technique for black-box optimization. However, its applicability is typically limited to moderate-budget problems due to the cubic complexity in computing the Gaussian process (GP) surrogate model. In large-budget scenarios, directly employing the standard GP model faces significant challenges in computational time and resource requirements. In this paper, we propose a novel approach, gradient-based sample selection Bayesian Optimization (GSSBO), to enhance the computational efficiency of BO. The GP model is constructed on a selected set of samples instead of the whole dataset. These samples are selected by leveraging gradient information to maintain diversity and representation. We provide a theoretical analysis of the gradient-based sample selection strategy and obtain explicit sublinear regret bounds for our proposed framework. Extensive experiments on synthetic and real-world tasks demonstrate that our approach significantly reduces the computational cost of GP fitting in BO while maintaining optimization performance comparable to baseline methods.

Problem

• Reducing computational cost of Gaussian process in Bayesian optimization
• Selecting diverse samples using gradient information for efficiency
• Maintaining optimization performance while lowering resource requirements

Innovation

• Gradient-based sample selection for efficiency
• Sublinear regret bounds theoretically proven
• Reduces GP fitting cost significantly