bayesian hyperparameter optimization

Efficient hyperparameter optimization for scale/precision parameters in stochastic models remains challenging under noisy evaluations. Method: This paper proposes a novel Bayesian optimization framework featuring a statistical surrogate model that enables closed-form analytical expressions of the expected acquisition function. Crucially, it derives, for the first time, a closed-form solution for the stochastic acquisition function optimizer—eliminating the need for Monte Carlo sampling. Contribution/Results: The method substantially reduces computational overhead in noisy environments. Evaluated on two computational engineering numerical experiments, it achieves up to a 40× improvement in iteration efficiency, while simultaneously reducing data requirements and total computational cost by approximately 40×, thereby significantly alleviating resource bottlenecks in hyperparameter tuning.

This work addresses the lack of convergence guarantees for non-Gaussian process (non-GP) surrogate models in Bayesian optimization (BO). To resolve this, we propose the first axiomatic pseudo-BO framework, formally characterizing the minimal conditions required for sequential black-box optimization to converge. Methodologically, we design a lightweight local-regression-based surrogate model coupled with a randomized prior mechanism for efficient uncertainty quantification, and integrate it with upper-confidence-bound-type acquisition strategies. Theoretically, we provide the first rigorous convergence analysis for non-GP BO. Empirically, our framework consistently outperforms state-of-the-art methods—including GP-BO, TuRBO, and ALEBO—across high-dimensional synthetic benchmarks, neural network hyperparameter tuning, and robot control tasks. Thus, it achieves both theoretical soundness and practical superiority.

Traditional Gaussian process-based Bayesian optimization (BO) struggles with black-box function optimization over discrete, combinatorial, and mixed-variable spaces due to its reliance on continuity and smoothness assumptions. Method: We propose Simulation-Driven Bayesian Optimization (SBBO), a novel paradigm that abandons explicit gradient-based optimization of surrogate models. Instead, SBBO relies solely on Markov Chain Monte Carlo (MCMC) posterior sampling and simulation-driven acquisition function evaluation, supporting arbitrary samplable Bayesian surrogates—including categorical GPs, discrete Bayesian networks, tree-augmented models, and deep generative models. Contribution/Results: SBBO is the first framework to systematically enable BO on combinatorial search spaces, offering strong generalizability and modular design. Experiments across diverse combinatorial optimization tasks demonstrate that SBBO significantly outperforms standard BO methods, validating its effectiveness, robustness, and model-agnosticism.

This work addresses the weak theoretical convergence guarantees for cumulative regret in Bayesian optimization (BO) under Gaussian noise. First, it establishes the first tight, pointwise prediction error bound for Gaussian process (GP) surrogate models in noisy settings. Leveraging this bound, the paper rigorously analyzes cumulative regret—within a frequentist framework—for acquisition functions including Upper Confidence Bound (UCB) and Thompson Sampling (TS), significantly tightening existing upper bounds for GP-UCB and GP-TS and yielding improved convergence rates. The analysis is further extended to Expected Improvement (EI) and related acquisition strategies, yielding a unified asymptotic convergence theory. The new error bound is broadly applicable, enabling principled theoretical performance evaluation and design of diverse noisy BO algorithms. Overall, this work provides a foundational theoretical tool for robust BO under observation noise.

This work addresses the challenge of implicit constraints—manifested as evaluation failures—arising from unreliable physics-based simulations in system architecture optimization. To tackle this, we propose a surrogate modeling framework that integrates probabilistic feasibility prediction with Bayesian optimization. Methodologically, we introduce a novel hybrid discrete Gaussian process to model the Probability of Validity (PoV), coupled with an interior-point selection strategy based on a minimum PoV threshold; the framework natively supports hierarchical design variables and multi-objective optimization. Our approach achieves the first successful solution for a jet engine architecture optimization task with a 50% simulation failure rate. Across multiple synthetic benchmarks and real-world case studies, it significantly improves convergence robustness and optimization success rate. The implementation is publicly available as the SBArchOpt Python library.

This study addresses the challenge that the performance of Bayesian optimization (BO) heavily depends on hyperparameter presets by proposing a data-driven bilevel BO framework for automatic parameter tuning. Methodologically, it introduces a novel pretraining paradigm that infers Gaussian processes from initial observations and generates sample paths. The outer level employs cumulative regret as the evaluation metric to automatically search for optimal hyperparameter configurations via Bayesian optimization. Experimental results demonstrate that this bilevel architecture efficiently identifies highly robust hyperparameter combinations from the candidate space, significantly reducing the cost of manual tuning while enhancing overall optimization efficiency.

This work addresses the challenge in Bayesian optimization where online tuning of Gaussian process kernel hyperparameters often leads to either poorly calibrated or overly conservative uncertainty estimates. To this end, it introduces a novel approach that formulates hyperparameter selection as a constrained online learning problem, dynamically balancing sharpness and calibration of uncertainty estimates along the optimization trajectory. The proposed method achieves adaptive control over uncertainty quality while preserving a sublinear regret bound. Empirical evaluations across multiple synthetic and real-world benchmarks demonstrate its superior performance: it consistently attains top-ranked final simple regret and maintains robust cumulative regret behavior throughout the optimization process.

In black-box optimization, the practical objective is often to efficiently identify a satisfactory solution that maintains performance under input perturbations encountered during deployment, rather than pursuing the global optimum. This work proposes a novel Bayesian optimization approach that explicitly distinguishes between controllable and uncontrollable perturbations during the optimization phase, thereby jointly modeling satisficing search and robustness under worst-case disturbances for the first time. By introducing a superlevel-set-oriented robustness metric and designing a corresponding acquisition function, the method efficiently identifies solutions that satisfy a prescribed performance threshold even under maximal perturbations. Empirical results demonstrate that this approach significantly outperforms existing methods that either solely seek optimality or neglect deployment-stage perturbations.

This work addresses the challenge of Bayesian optimization in mixed search spaces containing both continuous and non-uniformly spaced discrete variables, which commonly arise in natural sciences. Existing methods are hindered by the unavailability of gradients and the difficulty of optimizing acquisition functions over such domains. The authors propose the first extension of probabilistic reparameterization to non-equidistant discrete variables, enabling gradient-based optimization across the entire mixed space by integrating Gaussian process surrogates with tailored kernel functions. The method demonstrates robust performance on highly discontinuous and discretized objective functions, significantly improving sample efficiency over synthetic benchmarks and real-world scientific experiments. It is particularly well-suited for data-scarce, high-noise settings such as autonomous laboratories.

This work addresses the inaccuracy of lower-tail predictions in Gaussian processes (GPs) under Bayesian optimization, which arises from kernel and hyperparameter choices and undermines acquisition functions such as expected improvement. Focusing on the reliability of lower-tail predictions in noiseless settings, the paper introduces a target-oriented tail calibration framework featuring two novel concepts: spatial occurrence calibration and threshold μ-calibration. This framework establishes theoretical guarantees for predictive reliability in low-threshold regions and yields a post-processing method, termed tcGP, to enhance calibration quality. Empirical evaluations on standard benchmarks demonstrate that tcGP significantly outperforms both standard and globally calibrated GPs, improving lower-tail prediction accuracy, boosting Bayesian optimization performance, and ensuring that calibrated sampling points remain dense across the design space.