Hyperparameter Importance Analysis for Multi-Objective AutoML

2024-05-13

European Conference on Artificial Intelligence

AI Summary

In multi-objective AutoML, performance objectives—such as accuracy, inference latency, fairness, and energy consumption—are often conflicting, rendering conventional hyperparameter importance estimation unreliable. To address this, this paper proposes the first interpretable hyperparameter importance analysis framework specifically designed for multi-objective hyperparameter optimization. Our method integrates fANOVA with ablation paths to construct a surrogate model that enables prior-standardized quantification and dynamic importance assessment under user-specified multi-objective trade-offs. Extensive experiments across multiple benchmark datasets demonstrate that the framework significantly enhances both the interpretability of hyperparameter effects and the practical utility of hyperparameter tuning. It provides decision-makers with robust, quantitative insights into how individual hyperparameters influence competing objectives, thereby facilitating principled, transparent trade-off decisions in multi-objective AutoML workflows.

Technology Category

• Machine Learning: Auto ML and Hyperparameter Tuning
• Search and Optimization: Metareasoning and Metaheuristics
• Computer Vision: Interpretability, Explainability, and Transparency

Application Category

• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Hyperparameter optimization plays a pivotal role in enhancing the predictive performance and generalization capabilities of ML models. However, in many applications, we do not only care about predictive performance but also about additional objectives such as inference time, memory, or energy consumption. In such multi-objective scenarios, determining the importance of hyperparameters poses a significant challenge due to the complex interplay between the conflicting objectives. In this paper, we propose the first method for assessing the importance of hyperparameters in multi-objective hyperparameter optimization. Our approach leverages surrogate-based hyperparameter importance measures, i.e., fANOVA and ablation paths, to provide insights into the impact of hyperparameters on the optimization objectives. Specifically, we compute the a-priori scalarization of the objectives and determine the importance of the hyperparameters for different objective tradeoffs. Through extensive empirical evaluations on diverse benchmark datasets with three different objective pairs, each combined with accuracy, namely time, demographic parity loss, and energy consumption, we demonstrate the effectiveness and robustness of our proposed method. Our findings not only offer valuable guidance for hyperparameter tuning in multi-objective optimization tasks but also contribute to advancing the understanding of hyperparameter importance in complex optimization scenarios.

Problem

• Multi-objective Optimization
• Hyperparameter Importance
• Conflicting Performance Goals

Innovation

• Multi-objective Optimization
• Hyperparameter Importance
• Machine Learning Model Assessment