tune hyperparameters

This paper addresses the inefficiency and lack of scalability of manual hyperparameter tuning in large-scale machine learning. It systematically surveys hyperparameter optimization (HPO), unifying and classifying five mainstream paradigms: random/low-discrepancy search, bandit-based methods, Bayesian optimization, population-based (evolutionary) algorithms, and gradient-based differentiable optimization. The survey further extends to emerging settings—including online HPO, constrained HPO, and multi-objective HPO. Crucially, the work establishes novel theoretical connections between HPO and meta-learning as well as neural architecture search, yielding a comprehensive knowledge framework that articulates methodological principles, applicability boundaries, and inherent limitations. By clarifying the technical evolution and identifying key open challenges, this study provides a theoretically grounded yet practically actionable foundation for automated machine learning.

To address the exploration-exploitation imbalance and inefficient utilization of historical evaluation data in evolutionary algorithms (EAs) for hyperparameter optimization (HPO), this paper proposes an enhanced genetic algorithm (GA) framework integrated with a lightweight linear surrogate model. The linear surrogate is seamlessly embedded into GA’s selection and mutation operators, enabling a population-driven hybrid search strategy and performance-feedback-driven adaptive model updating—without requiring gradient computation or expensive modeling overhead. This design dynamically balances global exploration and local exploitation. On standard HPO benchmarks, the method achieves an average performance improvement of 1.89% (range: −3.45% to +6.55%) over state-of-the-art approaches, with negligible increase in training cost. Its core contribution lies in the first structured integration of linear surrogates with genetic operations, achieving a favorable trade-off among efficiency, accuracy, and scalability.

Parameter tuning for metaheuristic algorithms is notoriously complex and highly problem-dependent; existing online tuning methods suffer from limited dynamism and poor generalizability. To address this, we propose Clustering-based Parameter Adaptation (CPA), the first online parameter control framework for population-based algorithms that integrates unsupervised clustering into real-time parameter adaptation—requiring no prior knowledge, automatically identifying high-performing regions in parameter space, and generating nearby candidate parameters to enable structured exploration and adaptive evolution. CPA is embedded within a differential evolution framework and augmented with statistical significance testing. Comprehensive evaluation across high- and low-dimensional benchmark suites demonstrates that CPA significantly outperforms state-of-the-art automated parameter tuning methods in convergence speed, solution stability, and cross-dimensional generalizability, while exhibiting strong robustness and broad applicability across diverse optimization problems.

In Python search-based unit test generation, the DynaMOSA and MIO algorithms suffer from inefficient hyperparameter configurations; their default settings often yield suboptimal code coverage, while conventional grid search incurs excessive computational overhead. Method: This paper introduces differential evolution (DE) into the Pynguin framework for automated multi-objective hyperparameter optimization of search-based test generation algorithms. We design a fitness function and encoding scheme tailored to testing objectives and conduct end-to-end tuning experiments on standard benchmarks. Contribution/Results: DE-optimized DynaMOSA achieves significant improvements in branch and line coverage (average +8.2%) over baseline configurations. Moreover, it converges 3.7× faster than grid search while reducing tuning overhead by approximately 65%. This work establishes an efficient, reproducible hyperparameter optimization paradigm for search-based test generation.

This work addresses the challenge in random feature regression (RFR) that sampling-distribution hyperparameters are non-differentiable and thus inaccessible to gradient-based optimization. We propose a gradient-free black-box optimization framework based on ensemble Kalman inversion (EKI), the first application of EKI to learning hyperparameter distributions for random features. The method circumvents assumptions of objective differentiability and sample-wise gradient availability, enabling high-dimensional, robust, and fully automated hyperparameter tuning. By integrating random feature mapping with Bayesian modeling principles, our framework significantly improves RFR’s prediction accuracy and stability across diverse tasks—including global sensitivity analysis, chaotic system integration, and Bayesian inverse problems in atmospheric dynamics—demonstrating both broad applicability and practical utility.

This work addresses the long-standing challenge that reinforcement learning-based hyper-heuristics (RLHH) often struggle to effectively select low-level heuristics on standard benchmark functions. Focusing on the LeadingOnes problem and combining two randomized local search operators—RLS₁ and RLS₂—the study provides the first rigorous theoretical proof, supported by empirical validation, that RLHH can achieve the optimal expected runtime attainable by these two operators (up to lower-order terms) under appropriate parameter settings. This result refutes prior skepticism regarding RLHH’s capacity to learn effective heuristic selection policies. Furthermore, experimental results demonstrate that, at practical problem scales, RLHH outperforms the generalized randomized gradient hyper-heuristic, which also enjoys optimal theoretical runtime guarantees.

This work addresses the lack of statistical reliability guarantees in existing hyperparameter selection methods—such as grid search and Bayesian optimization—with respect to critical metrics like risk and safety. Building upon the learn-then-test (LTT) paradigm, the paper introduces a unified statistical framework that formulates hyperparameter selection as a multiple hypothesis testing problem, accommodating user-specified constraints on average risk, quantile risk, or information-theoretic measures. Leveraging tools from statistical inference—including p-values, e-values, and concentration inequalities—the method derives explicit, finite-sample bounds on error probabilities from first principles. This approach enables theoretically grounded validation and selection of hyperparameters, substantially enhancing the reliability and safety of AI systems in real-world deployment scenarios.

This work proposes a two-level deep reinforcement learning framework for large-scale Traveling Salesman Problems (TSP), wherein a recurrent Proximal Policy Optimization (PPO) agent dynamically controls both numerical and structural parameters of a genetic algorithm, enabling their decoupled analysis. The study provides the first empirical evidence that dynamic adjustment of structural parameters is crucial for avoiding premature convergence and escaping local optima, whereas numerical parameters serve only a fine-tuning role. Evaluated on large-scale TSP instances such as rl5915, the proposed method significantly outperforms static baselines, reducing the optimality gap by approximately 45%. These results offer a novel direction for automated algorithm design through adaptive parameter control in evolutionary computation.

This work addresses the high sensitivity of constraint programming solver performance to hyperparameter configurations and the prohibitive cost of manual tuning. The authors propose a resource-aware, two-phase auto-tuning framework that, within a limited time budget, first explores promising configurations and then solves the target problem using the best identified configuration. Innovatively integrating Bayesian optimization with Hamming distance-based search within a unified framework, the approach is implemented using CPMpy. Experimental evaluation on 114 combinatorial optimization instances demonstrates that the method outperforms the default configurations on 25.4% and 38.6% of instances for the ACE and Choco solvers, respectively, significantly surpassing either search strategy in isolation.

This work addresses the challenges in black-box optimization posed by heteroscedastic noise—namely, uncertain fitness evaluations, inaccurate solution ranking, and excessive computational cost—by introducing a confidence-driven dynamic sampling mechanism. The proposed approach integrates adaptive budget allocation and explicit averaging strategies within both CMA-ES and genetic algorithm frameworks. It further presents the first systematically constructed benchmark suite for heteroscedastic noise, thereby overcoming the conventional reliance on homoscedasticity and function smoothness assumptions. Experimental results demonstrate that the method consistently outperforms state-of-the-art algorithms under both homoscedastic and heteroscedastic noise conditions, achieving superior robustness and computational efficiency.