Algorithm Selection with Probing Trajectories: Benchmarking the Choice of Classifier Model

2025-01-20

AI Summary

This work addresses the algorithm selection problem in black-box optimization (BBO) based on probing trajectories. We systematically evaluate 17 time-series classifiers on the BBOB benchmark, using three types of short-horizon performance trajectories as inputs. Employing leave-one-instance and leave-one-problem cross-validation, we demonstrate—for the first time—that classifier architecture critically determines trajectory-driven algorithm selection performance. Specifically, feature-engineering-based models (e.g., TSF) and interval-based models (e.g., ROCKET) significantly outperform end-to-end sequence models such as RNNs and Transformers. The best-performing classifier achieves an average accuracy over 12 percentage points higher than LSTM and InceptionTime. Our study provides the first reproducible and interpretable guideline for selecting time-series classifiers in trajectory-based algorithm selection, establishing a foundation for principled, data-driven BBO meta-algorithm design.

Technology Category

• Search and Optimization: Algorithm Configuration
• Machine Learning: Time-Series/Data Streams
• Intelligent Robots: Learning & Optimization for ROB

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments

Abstract

Recent approaches to training algorithm selectors in the black-box optimisation domain have advocated for the use of training data that is algorithm-centric in order to encapsulate information about how an algorithm performs on an instance, rather than relying on information derived from features of the instance itself. Probing-trajectories that consist of a sequence of objective performance per function evaluation obtained from a short run of an algorithm have recently shown particular promise in training accurate selectors. However, training models on this type of data requires an appropriately chosen classifier given the sequential nature of the data. There are currently no clear guidelines for choosing the most appropriate classifier for algorithm selection using time-series data from the plethora of models available. To address this, we conduct a large benchmark study using 17 different classifiers and three types of trajectory on a classification task using the BBOB benchmark suite using both leave-one-instance out and leave-one-problem out cross-validation. In contrast to previous studies using tabular data, we find that the choice of classifier has a significant impact, showing that feature-based and interval-based models are the best choices.

Problem

• Sequence Data Classification
• Algorithm Selection
• Guideline Lack

Innovation

• Sequence Data Classification
• Feature-based Tools
• Time Interval-based Tools