Why Ranking Anomaly Detection Algorithms Isn't as Reliable as You May Think

2026-08-05

AI Summary

This study addresses the pronounced instability in performance rankings of anomaly detection algorithms under existing benchmarks, which undermines the reproducibility of conclusions. Leveraging 690 datasets from OddBench and seven representative algorithms, the authors conduct controlled multivariate experiments and statistical analyses to systematically evaluate how dataset selection, evaluation metrics, hyperparameters, and random seeds influence algorithm rankings. Their findings reveal that nearly every algorithm can emerge as “best” under specific configurations, with dataset choice and hyperparameter settings exerting the most significant impact. To quantify this instability, the paper introduces a novel metric and argues that reliable algorithm evaluation necessitates substantially larger and more diverse benchmark datasets.

Abstract

Anomaly detection is a safety-critical machine learning problem with applications ranging from fraud detection to network intrusion prevention and industrial monitoring. Despite the large number of proposed anomaly detection algorithms, many novel methods claim state-of-the-art performance. However, many authors do so under benchmark settings that are not aligned with one another. This lack of comparability raises concerns regarding the reproducibility and reliability of anomaly detection benchmarks. In this work, we study the impact of common benchmarking choices on the stability of algorithm rankings. Using seven representative anomaly detection algorithms and 690 datasets from the OddBench benchmark suite, we analyze how rankings change under varying dataset selections, evaluation metrics, hyperparameter configurations, and random seeds. To quantify this effect, we introduce a rank instability metric measuring the variability of algorithm rankings across benchmark settings. Our results show that algorithm rankings in anomaly detection are highly unstable. In many cases, almost every competitive algorithm can appear as the best-performing method under some benchmark configuration. Among the studied factors, dataset selection and hyperparameter choice contribute most strongly to ranking uncertainty, while random seeds and evaluation metrics have a comparatively limited impact. We also observe that reliable benchmarking requires substantially larger and more diverse dataset collections than the ones commonly used in prior work.

Problem

• anomaly detection
• benchmarking
• algorithm ranking
• reproducibility
• rank instability

Innovation

• rank instability
• anomaly detection benchmarking
• algorithm ranking stability
• hyperparameter sensitivity
• OddBench