Enhancing Trust-Region Bayesian Optimization via Newton Methods

2025-08-25

AI Summary

Bayesian optimization (BO) suffers from low sample efficiency in high-dimensional black-box optimization, while local trust-region methods (e.g., TuRBO) incur modeling bias and slow sampling due to reliance on localized Gaussian processes (GPs). Method: We propose a local quadratic modeling framework that integrates global GP gradient and Hessian information. Within multiple dynamically adjusted trust regions, we construct boundary-constrained second-order approximations and solve the resulting constrained quadratic programs via Newton’s method to efficiently select candidate points. This approach mitigates the vanishing-gradient issue of GPs in high dimensions while preserving modeling fidelity and heterogeneity. Contribution/Results: We provide theoretical convergence guarantees and demonstrate empirically—on both synthetic benchmarks and real-world tasks—that our method significantly outperforms state-of-the-art high-dimensional BO approaches, achieving faster convergence and enhanced optimization stability.

Technology Category

• Search and Optimization: Sampling/Simulation-based Search
• Reasoning under Uncertainty: Stochastic Optimization
• Machine Learning: Optimization

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Bayesian Optimization (BO) has been widely applied to optimize expensive black-box functions while retaining sample efficiency. However, scaling BO to high-dimensional spaces remains challenging. Existing literature proposes performing standard BO in multiple local trust regions (TuRBO) for heterogeneous modeling of the objective function and avoiding over-exploration. Despite its advantages, using local Gaussian Processes (GPs) reduces sampling efficiency compared to a global GP. To enhance sampling efficiency while preserving heterogeneous modeling, we propose to construct multiple local quadratic models using gradients and Hessians from a global GP, and select new sample points by solving the bound-constrained quadratic program. Additionally, we address the issue of vanishing gradients of GPs in high-dimensional spaces. We provide a convergence analysis and demonstrate through experimental results that our method enhances the efficacy of TuRBO and outperforms a wide range of high-dimensional BO techniques on synthetic functions and real-world applications.

Problem

• Enhancing Bayesian Optimization in high-dimensional spaces
• Improving sampling efficiency via local quadratic models
• Addressing vanishing gradient issues in Gaussian Processes

Innovation

• Multiple local quadratic models from global GP gradients
• Bound-constrained quadratic programming for sample selection
• Addressing vanishing GP gradients in high-dimensional spaces