A Balanced Approach of Rapid Genetic Exploration and Surrogate Exploitation for Hyperparameter Optimization

2025-04-10

IEEE Access

AI Summary

To address the exploration-exploitation imbalance and inefficient utilization of historical evaluation data in evolutionary algorithms (EAs) for hyperparameter optimization (HPO), this paper proposes an enhanced genetic algorithm (GA) framework integrated with a lightweight linear surrogate model. The linear surrogate is seamlessly embedded into GA’s selection and mutation operators, enabling a population-driven hybrid search strategy and performance-feedback-driven adaptive model updating—without requiring gradient computation or expensive modeling overhead. This design dynamically balances global exploration and local exploitation. On standard HPO benchmarks, the method achieves an average performance improvement of 1.89% (range: −3.45% to +6.55%) over state-of-the-art approaches, with negligible increase in training cost. Its core contribution lies in the first structured integration of linear surrogates with genetic operations, achieving a favorable trade-off among efficiency, accuracy, and scalability.

Technology Category

• Search and Optimization: Evolutionary Computation
• Machine Learning: Evolutionary Learning
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

This paper introduces a novel approach to hyperparameter optimization (HPO), proposing a methodology that balances exploration and exploitation to enhance optimization performance. While evolutionary algorithms (EAs) have shown potential in HPO, they often struggle with effective exploitation. To address this limitation, we propose an improved hyperparameter optimization (HPO) framework that integrates a linear surrogate model into the genetic algorithm (GA). The GA’ss flexible structure allows for seamless integration of multiple optimization strategies, and the surrogate model significantly boosts its exploitation capabilities. Specifically, we achieved an average performance improvement of 1.89% (max 6.55%, min −3.45%) over the existing state-of-the-art HPO strategy.

Problem

• Balancing exploration and exploitation in hyperparameter optimization
• Improving exploitation in evolutionary algorithms for HPO
• Integrating surrogate models with genetic algorithms for better HPO

Innovation

• Balances genetic exploration and surrogate exploitation
• Integrates linear surrogate model into GA
• Improves HPO performance by 1.89%