particle swarm optimization

Efficient hyperparameter optimization for scale/precision parameters in stochastic models remains challenging under noisy evaluations. Method: This paper proposes a novel Bayesian optimization framework featuring a statistical surrogate model that enables closed-form analytical expressions of the expected acquisition function. Crucially, it derives, for the first time, a closed-form solution for the stochastic acquisition function optimizer—eliminating the need for Monte Carlo sampling. Contribution/Results: The method substantially reduces computational overhead in noisy environments. Evaluated on two computational engineering numerical experiments, it achieves up to a 40× improvement in iteration efficiency, while simultaneously reducing data requirements and total computational cost by approximately 40×, thereby significantly alleviating resource bottlenecks in hyperparameter tuning.

This work addresses the lack of provable joint guarantees on generalization and convergence in learned optimization algorithms. Methodologically: (1) it establishes the first PAC-Bayesian generalization bound for unbounded losses, leveraging exponential-family posterior distributions; (2) it formulates optimizer learning as a tractable one-dimensional global optimization problem—convex or non-convex—whose solution is analytically characterizable; and (3) it integrates stochastic optimization design with rigorous theoretical analysis to explicitly trade off convergence rate against generalization error. Empirically, the learned optimizers achieve order-of-magnitude improvements over state-of-the-art methods across four diverse real-world tasks—including neural architecture search, meta-learning, adversarial training, and federated learning—while all gains are underpinned by formal theoretical guarantees. This constitutes the first learning-to-optimize framework endowed with a provably tight PAC-Bayesian generalization bound and jointly certified convergence–generalization performance.

In Bayesian optimization (BO), Expected Improvement (EI) and its variants frequently suffer from numerical ill-conditioning—especially in high-dimensional, constrained, parallel, or noisy settings—leading to unstable performance and optimization failure. This work systematically identifies a shared numerical instability mechanism across EI, Expected Hypervolume Improvement (EHVI), and their extensions. To address this, we propose LogEI, a novel family of acquisition functions that applies a logarithmic transformation to achieve lossless reformulation: the global optimum is preserved exactly, while numerical robustness is significantly enhanced. We further derive closed-form analytical gradients and integrate them with efficient optimization strategies to guarantee stable convergence. Evaluated across diverse benchmarks—including constrained, multi-objective, parallel, and noisy BO tasks—LogEI consistently outperforms standard EI variants and matches or exceeds state-of-the-art methods. Our results empirically demonstrate that numerical robustness is a decisive factor for practical BO efficacy.

To address the high computational cost and curse of dimensionality in high-dimensional Bayesian optimization, this paper proposes the Expected Coordinate Improvement (ECI) acquisition function—the first to incorporate coordinate descent into acquisition design: at each iteration, ECI greedily optimizes along a single coordinate axis, circumventing full-space search. We prove ECI’s consistency theoretically. Practically, ECI integrates a Gaussian process surrogate, coordinate-aligned gradient approximation, randomized coordinate selection, and Monte Carlo estimation for efficient evaluation. On 100-dimensional benchmark functions, ECI achieves an 8.2× speedup over Expected Improvement (EI) and GP-UCB, while improving simple regret convergence by 37%. Its efficacy and practicality are further validated on neural network hyperparameter tuning tasks.

This work addresses the family of parametric optimization problems and proposes the first unified, data-driven framework for analyzing the generalization performance of both classical and learned optimizers. Methodologically: (1) it introduces PAC-Bayes theory to the analysis of learned optimizers, deriving verifiable, high-probability generalization upper bounds; (2) it establishes performance bounds for classical optimizers based on empirical convergence rates; and (3) it pioneers a learning paradigm that directly minimizes the PAC-Bayes bound during training. Evaluated on signal processing, control, and meta-learning tasks, the derived bounds are significantly tighter than conventional worst-case guarantees. Moreover, the theoretical generalization guarantees for learned optimizers consistently exceed the empirical performance of their non-learned baselines—thereby unifying theoretical rigor with practical efficacy.

This work addresses the unreliability of decisions in generative-model-driven stochastic optimization, which arises from sampler misspecification and limited simulation budgets. To tackle this issue, the paper introduces a Sampler-Robust Optimization (SRO) framework that, for the first time, incorporates robust optimization at the sampler level. SRO explicitly models the worst-case sampler by perturbing the learned generator and integrates a minimax optimization strategy with a sharpness-aware robustness mechanism. The approach is applicable to both density-explicit and implicit generative models. Theoretical analysis provides high-probability upper bounds on performance, and empirical results demonstrate that SRO significantly improves out-of-sample performance and decision stability in portfolio optimization tasks.

Black-box variational inference heavily relies on optimizer hyperparameter tuning, hindering truly plug-and-play Bayesian inference. To address this, this work presents the first systematic evaluation of 56 adaptive stochastic gradient optimizers across 1,092 Bayesian inference problems, spanning diverse posterior dimensionalities, condition numbers, and variational families. Based on over 550,000 optimization runs (approximately 15 CPU-years of computation), the study finds that no single optimizer consistently dominates, yet a carefully selected ensemble of just five algorithms reliably approximates the best-performing optimizer across tasks. These findings yield a practical strategy and strong baseline for tuning-free variational inference and establish a new benchmark for future algorithmic development and evaluation.

This work addresses stochastic convex optimization in the presence of both additive and multiplicative noise, where conventional methods such as sample average approximation suffer significant performance degradation under finite-sample regimes. We propose VISOR, a novel algorithm that integrates variance reduction with acceleration mechanisms, achieving instance-optimal performance that matches the information-theoretic local minimax lower bound for the first time under finite samples. Through non-asymptotic analysis, we establish the sharpest known instance-dependent generalization error bounds for generalized linear models—including linear regression—while simultaneously attaining optimal sample complexity and oracle complexity.

This work addresses the challenge of Bayesian optimization in mixed search spaces containing both continuous and non-uniformly spaced discrete variables, which commonly arise in natural sciences. Existing methods are hindered by the unavailability of gradients and the difficulty of optimizing acquisition functions over such domains. The authors propose the first extension of probabilistic reparameterization to non-equidistant discrete variables, enabling gradient-based optimization across the entire mixed space by integrating Gaussian process surrogates with tailored kernel functions. The method demonstrates robust performance on highly discontinuous and discretized objective functions, significantly improving sample efficiency over synthetic benchmarks and real-world scientific experiments. It is particularly well-suited for data-scarce, high-noise settings such as autonomous laboratories.

This work addresses the challenges of traditional Bayesian optimization, which suffers from cubic computational complexity and difficulties in adapting global surrogate models to local optimization needs. The authors propose a novel approach that, for the first time, integrates recursive binary space partitioning into the Bayesian optimization framework. By jointly adapting Gaussian process modeling and acquisition strategies, the method achieves an adaptive balance between exploration and exploitation. This design reduces computational complexity from cubic to linear while maintaining high optimization performance. Empirical evaluations on seven standard benchmark functions spanning 6 to 124 dimensions demonstrate that the proposed method consistently outperforms state-of-the-art Bayesian optimization libraries, achieving superior efficiency and solution quality.