tune focal loss

This paper addresses the inefficiency and lack of scalability of manual hyperparameter tuning in large-scale machine learning. It systematically surveys hyperparameter optimization (HPO), unifying and classifying five mainstream paradigms: random/low-discrepancy search, bandit-based methods, Bayesian optimization, population-based (evolutionary) algorithms, and gradient-based differentiable optimization. The survey further extends to emerging settings—including online HPO, constrained HPO, and multi-objective HPO. Crucially, the work establishes novel theoretical connections between HPO and meta-learning as well as neural architecture search, yielding a comprehensive knowledge framework that articulates methodological principles, applicability boundaries, and inherent limitations. By clarifying the technical evolution and identifying key open challenges, this study provides a theoretically grounded yet practically actionable foundation for automated machine learning.

In automated performance tuning, optimizer hyperparameters have long been overlooked, and their impact on overall tuning efficacy remains systematically uninvestigated. Method: This paper introduces the novel paradigm of “tuner hyperparameter optimization” to address efficiency bottlenecks caused by suboptimal hyperparameter configurations. We design a robust cross-search-space evaluation protocol, construct a reproducible benchmark dataset and open-source toolkit, and incorporate a low-cost simulation replay mechanism to enable efficient meta-strategy optimization—fully adhering to FAIR (Findable, Accessible, Interoperable, Reusable) principles. Contribution/Results: Experiments demonstrate that lightweight hyperparameter tuning improves tuner performance by 94.8% on average; integrating meta-strategy optimization further boosts average gain to 204.7%. This work establishes both theoretical foundations and practical pathways for self-enhancement in automated tuning frameworks.

This paper identifies a systemic issue in machine learning: preprocessing hyperparameters—such as missing-value imputation strategies—are frequently overlooked yet substantially bias model evaluation. Current practice often involves informal, post-hoc tuning of preprocessing steps, leading to optimistic performance estimates and irreproducible results. To address this, the authors formally distinguish and empirically analyze the coupling effects between algorithmic and preprocessing hyperparameters. Using a modular supervised learning workflow model, controlled variable experiments, replication of canonical case studies, and bias diagnostics, they quantify the resulting optimistic bias. Key contributions include: (1) establishing preprocessing hyperparameters as equally critical as algorithmic ones; (2) proposing formal modeling principles to eliminate informal preprocessing tuning; and (3) delivering actionable reporting guidelines for ML practitioners, thereby significantly enhancing model credibility and reproducibility.

This study investigates whether tuning hyperparameters on test sets severely compromises the reliability of model evaluation and benchmark rankings. Through systematic experimental designs across multi-task benchmarks including MNIST, CIFAR, and GLUE, combined with statistical significance testing and ranking stability analysis, this work quantifies the actual impact of such practices. Challenging the conventional dogma that strictly prohibits test set tuning, the findings demonstrate that while this practice induces slight performance inflation, its magnitude frequently remains below the level of random noise and does not alter the relative ordering of models. By providing empirical evidence for re-examining this long-standing convention, this research advocates for a more open and transparent paradigm in evaluation reporting.

Deep neural networks often suffer from miscalibration—exhibiting overconfidence or underconfidence—leading to unreliable probabilistic predictions, particularly critical in safety-sensitive domains like healthcare. Method: We propose Focal Calibration Loss (FCL), the first unified calibration framework that rigorously embeds Euclidean distance constraints into a strictly proper scoring rule, preserving Focal Loss’s sensitivity to hard examples while guaranteeing theoretically grounded probability calibration. Our approach introduces a novel loss function based on Euclidean norm minimization, grounded in strict propriety analysis, and adapts it to medical models (e.g., CheXNet) with end-to-end web deployment validation. Results: Extensive experiments across multiple architectures and datasets demonstrate that FCL simultaneously achieves new state-of-the-art performance in both Expected Calibration Error (ECE) and classification accuracy. It significantly enhances the reliability and trustworthiness of medical AI systems without compromising predictive performance.

This work addresses the lack of provable generalization guarantees for multidimensional hyperparameter tuning in complex, non-smooth spaces. It proposes the first data-driven framework for such settings, establishing generalization bounds under mild assumptions by integrating structured loss with validation loss. Leveraging tools from real algebraic geometry, the analysis characterizes the complexity of semi-algebraic function classes, yielding tighter and more broadly applicable generalization bounds. The framework’s effectiveness and learnability are demonstrated on models such as weighted group Lasso and weighted fused Lasso, offering both theoretical foundations and practical methodologies for multidimensional hyperparameter optimization.

Hyperparameter tuning suffers from high computational cost and lacks interpretable guidance regarding parameter importance ranking, pairwise interactions, and critical value ranges. To address this, we propose MetaSHAP—the first framework integrating SHAP value analysis with meta-learning. Leveraging over 9 million historical machine learning pipelines, MetaSHAP models the directional impact, pairwise interactions, and sensitivity intervals of hyperparameters, generating dataset- and algorithm-specific, interpretable tuning recommendations. Our method combines surrogate model construction, Bayesian optimization guidance, and large-scale benchmarking across 164 classification datasets and 14 classifiers. Experiments demonstrate that MetaSHAP yields reliable hyperparameter importance estimates and guides Bayesian optimization to state-of-the-art performance, significantly overcoming the limitations of conventional black-box tuning approaches.

This work addresses the lack of interpretability in existing methods regarding the influence of hyperparameters in multi-objective optimization. The authors propose a novel game-theoretic framework that, for the first time, integrates Shapley effects with the Pareto front to enable objective-aware global sensitivity analysis, thereby uncovering key hyperparameters and their interactions under different optimization objectives. This approach not only identifies efficient hyperparameter configurations and substantially reduces the search space but also facilitates early-stage model performance estimation. The effectiveness and generalizability of the framework are empirically validated across three distinct neural network architectures and tasks.

This work addresses the mismatch between conventional machine learning practices and the specific performance requirements of clinical tasks in healthcare settings. Traditional approaches rely on differentiable validation losses for model optimization, which often fail to align with clinically meaningful outcomes. To bridge this gap, the paper proposes replacing standard loss functions with non-differentiable yet clinically interpretable custom metrics to guide critical optimization decisions—such as hyperparameter selection and training termination—thereby redefining the model validation pipeline. In two controlled experiments, models optimized using this framework demonstrated significantly superior performance on key clinical tasks compared to those guided by conventional differentiable validation losses. This approach overcomes the inherent limitation of relying solely on differentiable objectives and better aligns medical AI development with real-world clinical goals.

This study addresses the asymmetric cost of early versus late errors in remaining useful life (RUL) prediction for predictive maintenance, where conventional single-objective optimization struggles to balance directional bias. We propose a multi-objective hyperparameter optimization framework that treats the optimization objective itself as a design variable, jointly optimizing R² and the NASA scoring function. The NSGA-II algorithm is combined with an entropy weight–CRITIC method to resolve conflicting objectives, and the approach is systematically evaluated across five architectures: MLP, LSTM, XGBoost, TCN, and Transformer. Experiments on the C-MAPSS dataset demonstrate that the proposed method reduces directional imbalance by 33%. Furthermore, the results reveal that model rankings are highly sensitive to hyperparameter configurations and quantify cross-domain generalization gaps, offering practical insights for reliable RUL estimation.