VUS: Effective and Efficient Accuracy Measures for Time-Series Anomaly Detection

2025-02-18

AI Summary

Traditional point-wise evaluation metrics (e.g., Precision/Recall) for time-series anomaly detection (AD) suffer from two key limitations: poor characterization of interval-based anomalies and high sensitivity to threshold selection. To address these, this paper proposes a novel, threshold-free, parameter-free evaluation framework. Methodologically, it first establishes the theoretical advantages of AUC-style metrics for time-series AD; then introduces the Volume Under the Surface (VUS) family of metrics, which jointly assesses anomaly intervals and multi-dimensional detection parameters via surface integration; finally, it designs two efficient numerical approximation algorithms. Experimental results demonstrate that the four VUS variants exhibit significantly superior robustness over state-of-the-art methods under noise perturbation, localization shift, and varying anomaly density—while achieving computational speedups of several orders of magnitude.

Technology Category

• Machine Learning: Evaluation and Analysis
• Data Mining & Knowledge Management: Anomaly/Outlier Detection
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs

Abstract

Anomaly detection (AD) is a fundamental task for time-series analytics with important implications for the downstream performance of many applications. In contrast to other domains where AD mainly focuses on point-based anomalies (i.e., outliers in standalone observations), AD for time series is also concerned with range-based anomalies (i.e., outliers spanning multiple observations). Nevertheless, it is common to use traditional point-based information retrieval measures, such as Precision, Recall, and F-score, to assess the quality of methods by thresholding the anomaly score to mark each point as an anomaly or not. However, mapping discrete labels into continuous data introduces unavoidable shortcomings, complicating the evaluation of range-based anomalies. Notably, the choice of evaluation measure may significantly bias the experimental outcome. Despite over six decades of attention, there has never been a large-scale systematic quantitative and qualitative analysis of time-series AD evaluation measures. This paper extensively evaluates quality measures for time-series AD to assess their robustness under noise, misalignments, and different anomaly cardinality ratios. Our results indicate that measures producing quality values independently of a threshold (i.e., AUC-ROC and AUC-PR) are more suitable for time-series AD. Motivated by this observation, we first extend the AUC-based measures to account for range-based anomalies. Then, we introduce a new family of parameter-free and threshold-independent measures, Volume Under the Surface (VUS), to evaluate methods while varying parameters. We also introduce two optimized implementations for VUS that reduce significantly the execution time of the initial implementation. Our findings demonstrate that our four measures are significantly more robust in assessing the quality of time-series AD methods.

Problem

• Evaluates accuracy measures for time-series anomaly detection.
• Extends AUC-based measures for range-based anomalies.
• Introduces Volume Under the Surface (VUS) measures.

Innovation

• Extends AUC measures for range-based anomalies
• Introduces Volume Under the Surface (VUS) measures
• Optimizes VUS for reduced execution time