Accuracy Can Lie: On the Impact of Surrogate Model in Configuration Tuning

2025-01-03

AI Summary

This study challenges the prevailing assumption that higher surrogate model accuracy invariably yields better configuration tuning performance. Method: We conduct a large-scale empirical investigation—13,612 experiments over 13 months across 10 model families, 17 tuners, and 29 systems—introducing a fitness-landscape-based attribution framework. We employ multidimensional error metrics (e.g., RMSE, MAE), cross-system benchmarking, and statistical significance testing to rigorously quantify tuning outcomes. Contribution/Results: We are the first to empirically demonstrate a non-monotonic relationship between surrogate accuracy and tuning quality: up to 58% of cases show no improvement, and 24% exhibit statistically significant degradation. We identify widespread suboptimal surrogate selection and reveal that accuracy thresholds dynamically shift with error regimes. Our findings yield a reproducible tuner–surrogate matching guide and marginal accuracy-benefit analysis, shifting the surrogate selection paradigm from “higher accuracy is better” to “fitness-aware adaptivity.”

Technology Category

Search and Optimization: Sampling/Simulation-based SearchMachine Learning: Calibration & Uncertainty QuantificationReasoning under Uncertainty: Stochastic Optimization

Application Category

User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating successSearch and Retrieval-Augmented AI: Web evaluation methodologies and metricsEconomics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

To ease the expensive measurements during configuration tuning, it is natural to build a surrogate model as the replacement of the system, and thereby the configuration performance can be cheaply evaluated. Yet, a stereotype therein is that the higher the model accuracy, the better the tuning result would be. This"accuracy is all"belief drives our research community to build more and more accurate models and criticize a tuner for the inaccuracy of the model used. However, this practice raises some previously unaddressed questions, e.g., Do those somewhat small accuracy improvements reported in existing work really matter much to the tuners? What role does model accuracy play in the impact of tuning quality? To answer those related questions, we conduct one of the largest-scale empirical studies to date-running over the period of 13 months 24*7-that covers 10 models, 17 tuners, and 29 systems from the existing works while under four different commonly used metrics, leading to 13,612 cases of investigation. Surprisingly, our key findings reveal that the accuracy can lie: there are a considerable number of cases where higher accuracy actually leads to no improvement in the tuning outcomes (up to 58% cases under certain setting), or even worse, it can degrade the tuning quality (up to 24% cases under certain setting). We also discover that the chosen models in most proposed tuners are sub-optimal and that the required % of accuracy change to significantly improve tuning quality varies according to the range of model accuracy. Deriving from the fitness landscape analysis, we provide in-depth discussions of the rationale behind, offering several lessons learned as well as insights for future opportunities. Most importantly, this work poses a clear message to the community: we should take one step back from the natural"accuracy is all"belief for model-based configuration tuning.

Problem

• Model-based System Tuning
• Model Accuracy Impact
• Precision-Effectiveness Correlation

Innovation

• Model-based Optimization
• Accuracy-Effectiveness Discrepancy
• Perspective on Model Selection