When to Use Which? Benchmarking Optimisers for Configurable Systems under Varying Budgets

2026-07-17

AI Summary

This study addresses the lack of systematic guidance for selecting optimizers in software configuration tuning under varying time budgets. Through a large-scale empirical evaluation across 22 real-world configurable systems, the authors compare eight prominent optimization approaches, including model-based methods such as SMAC and model-free techniques like genetic algorithms. The results reveal that model-based optimizers generally outperform others under low time budgets, whereas model-free methods dominate when budgets are high. Notably, FLASH demonstrates consistently strong performance across all budget regimes. This robustness stems from the prevalence of high-quality local optima with large basins of attraction in most systems, offering practitioners a reliable, budget-agnostic choice for configuration optimization.

Abstract

Software configuration tuning is crucial for optimising system performance, and various optimisers have emerged over the last decade. Yet, the time required during the tuning process may vary across systems. In some systems (e.g., PostgreSQL), it may take a few minutes to measure a configuration, whereas in some others (e.g., MariaDB), it can take several hours. Moreover, even within the same system, users may have varying budgets and preferred settings. This naturally raises a question -- Given a budget level, which optimiser is the best choice for SE practitioners? This matters because optimisers usually have their own ``comfort zone'' and may perform very differently under distinct budgets. In this paper, we aim to answer this question. We systematically evaluate eight well-established optimisers across 22 configurable systems under varying budget levels. We find that, unsurprisingly, model-based optimisers (e.g., SMAC) are well-suited under tight budgets, and model-free optimisers (e.g., GAs) become superior with more generous budgets. However, interestingly, there is one optimiser, FLASH, that performs consistently well on most systems regardless of budgets. We lastly investigate the reasons behind this phenomenon and find that many systems possess good local optima (with large basins of attraction), allowing greedy optimisers (e.g., FLASH) to achieve strong performance. Source code, data, and supplementary materials of this work are available at https://anonymous.4open.science/r/Config-W2W-98B2.

Problem

• optimiser selection
• configurable systems
• budget constraints
• software configuration tuning
• performance optimisation

Innovation

• optimizer benchmarking
• configurable systems
• budget-aware tuning
• FLASH
• local optima