hyperparameter tuning

In automated performance tuning, optimizer hyperparameters have long been overlooked, and their impact on overall tuning efficacy remains systematically uninvestigated. Method: This paper introduces the novel paradigm of “tuner hyperparameter optimization” to address efficiency bottlenecks caused by suboptimal hyperparameter configurations. We design a robust cross-search-space evaluation protocol, construct a reproducible benchmark dataset and open-source toolkit, and incorporate a low-cost simulation replay mechanism to enable efficient meta-strategy optimization—fully adhering to FAIR (Findable, Accessible, Interoperable, Reusable) principles. Contribution/Results: Experiments demonstrate that lightweight hyperparameter tuning improves tuner performance by 94.8% on average; integrating meta-strategy optimization further boosts average gain to 204.7%. This work establishes both theoretical foundations and practical pathways for self-enhancement in automated tuning frameworks.

This paper addresses the inefficiency and lack of scalability of manual hyperparameter tuning in large-scale machine learning. It systematically surveys hyperparameter optimization (HPO), unifying and classifying five mainstream paradigms: random/low-discrepancy search, bandit-based methods, Bayesian optimization, population-based (evolutionary) algorithms, and gradient-based differentiable optimization. The survey further extends to emerging settings—including online HPO, constrained HPO, and multi-objective HPO. Crucially, the work establishes novel theoretical connections between HPO and meta-learning as well as neural architecture search, yielding a comprehensive knowledge framework that articulates methodological principles, applicability boundaries, and inherent limitations. By clarifying the technical evolution and identifying key open challenges, this study provides a theoretically grounded yet practically actionable foundation for automated machine learning.

This paper identifies a systemic issue in machine learning: preprocessing hyperparameters—such as missing-value imputation strategies—are frequently overlooked yet substantially bias model evaluation. Current practice often involves informal, post-hoc tuning of preprocessing steps, leading to optimistic performance estimates and irreproducible results. To address this, the authors formally distinguish and empirically analyze the coupling effects between algorithmic and preprocessing hyperparameters. Using a modular supervised learning workflow model, controlled variable experiments, replication of canonical case studies, and bias diagnostics, they quantify the resulting optimistic bias. Key contributions include: (1) establishing preprocessing hyperparameters as equally critical as algorithmic ones; (2) proposing formal modeling principles to eliminate informal preprocessing tuning; and (3) delivering actionable reporting guidelines for ML practitioners, thereby significantly enhancing model credibility and reproducibility.

To address the inefficiency and poor generalizability of manual hyperparameter tuning—particularly for learning rates—this paper proposes a dynamic online meta-optimization framework that formulates learning rate adaptation as a discounted cumulative regret minimization problem over time. The method employs a gradient-based meta-update mechanism, enabling plug-and-play integration with any first-order optimizer (e.g., SGD, Adam) to achieve decoupled, real-time, adaptive step-size optimization. Key contributions include: (i) the first formalization of meta-optimization as discounted regret minimization; and (ii) a low-complexity variant that preserves theoretical rigor while ensuring computational efficiency and strong generalization. Experiments across diverse tasks demonstrate faster convergence, enhanced robustness to initialization and task heterogeneity, competitive performance against hand-tuned optimal schedulers, and significantly lower computational overhead compared to conventional hyperparameter search methods.

The theoretical mechanisms underlying parameter-efficient fine-tuning (PEFT) methods for large pre-trained models remain poorly understood, and the performance disparities among existing approaches lack principled explanations. Method: This paper establishes, for the first time, a unified theoretical framework grounded in matrix decomposition, revealing that diverse PEFT methods fundamentally perform optimization under low-rank constraints. Leveraging this insight, we propose two novel PEFT methods and a general-purpose enhancement framework—designed with theoretical rigor and architectural generality—through SVD- and LoRA-style modeling analysis, modular design, and multi-task empirical validation. Contribution/Results: Our approach significantly improves the performance of canonical PEFT methods—including LoRA and Adapter—across mainstream NLP benchmarks. This work provides the first principle-level, systematic explanation of PEFT and establishes an extensible technical pathway for future advancements.

Hyperparameter tuning suffers from high computational cost and lacks interpretable guidance regarding parameter importance ranking, pairwise interactions, and critical value ranges. To address this, we propose MetaSHAP—the first framework integrating SHAP value analysis with meta-learning. Leveraging over 9 million historical machine learning pipelines, MetaSHAP models the directional impact, pairwise interactions, and sensitivity intervals of hyperparameters, generating dataset- and algorithm-specific, interpretable tuning recommendations. Our method combines surrogate model construction, Bayesian optimization guidance, and large-scale benchmarking across 164 classification datasets and 14 classifiers. Experiments demonstrate that MetaSHAP yields reliable hyperparameter importance estimates and guides Bayesian optimization to state-of-the-art performance, significantly overcoming the limitations of conventional black-box tuning approaches.

Online learning systems face prohibitively high hyperparameter search costs due to persistent data distribution shifts. This paper proposes a two-stage efficient hyperparameter search paradigm tailored for non-stationary sequential data: Stage I employs lightweight data summarization and sequence forecasting to rapidly identify high-potential configurations; Stage II performs full training only on the shortlisted candidates. Departing from conventional performance-maximization search strategies, our approach prioritizes early, accurate pruning—significantly reducing redundant computation. Evaluated on the Criteo 1TB dataset, it achieves up to 10× reduction in search cost. Its efficacy and generalizability are further validated in large-scale industrial advertising systems. The core innovation lies in reframing hyperparameter optimization—from static performance tuning to dynamic, adaptivity-driven candidate screening—thereby overcoming the fundamental limitations of traditional methods in time-varying environments.

This study investigates whether default hyperparameters in machine learning libraries serve as effective initial points for Bayesian optimization to accelerate convergence. The authors conduct the first large-scale empirical evaluation by initializing optimization with samples drawn from a truncated Gaussian distribution centered around default values and comparing this strategy against uniform random initialization. Experiments span three optimization frameworks—BoTorch, Optuna, and Scikit-Optimize—combined with Random Forest, SVM, and MLP models across five standard datasets. Results show that default hyperparameters do not yield statistically significant performance improvements (p = 0.141–0.908), and any early advantage they confer dissipates as optimization progresses. These findings suggest that default values lack informative prior knowledge, challenging the common heuristic of using them as starting points in hyperparameter optimization.

Large-scale hyperparameter tuning is computationally expensive and lacks rigorous theoretical foundations. Method: This paper introduces the “trajectory invariance” principle, demonstrating that coupling learning rate and weight decay induces near-identical training loss curves, gradient noise profiles, and gradient norm dynamics across diverse hyperparameter configurations—effectively collapsing the two-dimensional tuning space into a one-dimensional manifold. Contribution/Results: This principle establishes the first universal guiding principle for hyperparameter optimization, substantially reducing search dimensionality and tuning cost. It revises existing scaling laws and challenges conventional assumptions—such as independent tuning of learning rate and weight decay. Validated across multiple architectures and tasks, the principle is grounded in pretraining loss analysis, gradient noise modeling, and empirical gradient norm observation, confirming its broad applicability and practical utility.

This study addresses the high cost and sensitivity of large language model fine-tuning to data quality and hyperparameters, highlighting the need for pre-training performance prediction. The authors propose TuneAhead, a novel framework that enables accurate forecasting of fine-tuning outcomes by constructing a lightweight regression model from static data descriptors and dynamic features derived from short, standardized probing runs. Integrated SHAP analysis provides interpretable diagnostics to support informed “proceed/abort” decisions prior to full-scale training. Evaluated across 370 hold-out tests, TuneAhead achieves an RMSE of 1.47 percentage points, with 95.1% of predictions falling within ±3 percentage points of ground truth—significantly outperforming baseline methods such as Early-Stop Extrapolation and ProxyLM.