Adaptive Anomaly Detection in the Presence of Concept Drift: Extended Report

2025-06-18

AI Summary

In dynamic time series, the co-occurrence of concept drift and anomalies impedes their discrimination, leading to false detections, error propagation, and excessive model updates. This paper proposes AnDri—the first unified framework for joint concept drift detection and anomalous subsequence identification. Its key contributions are: (1) Adjacency-aware Hierarchical Clustering (AHC), which preserves temporal locality to precisely segment evolving normal patterns; (2) an adaptive normal-pattern update mechanism integrated with sliding-window subsequence modeling; and (3) a joint drift–anomaly discrimination strategy. Evaluated on diverse real-world time series datasets, AnDri achieves an average 12.7% improvement in F1-score and reduces spurious model update rate by 63%. By mitigating error accumulation, it significantly enhances robustness for downstream time-series analysis tasks.

Technology Category

• Machine Learning: Time-Series/Data Streams
• Data Mining & Knowledge Management: Anomaly/Outlier Detection
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Web Mining and Content Analysis: Models for Web evolution
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Data changes to reflect evolving user behaviour, preferences, and changes in the environment. Such changes may occur due to expected shifts in the data distribution, i.e., concept drift, or unexpected anomalous changes. The presence of concept drift poses challenges for anomaly detection in time series. While anomalies are caused by undesirable changes in the data, differentiating abnormal changes from varying normal behaviours is difficult due to differing frequencies of occurrence, varying time intervals when normal patterns occur. Differentiating between concept drift and anomalies is critical for accurate analysis as studies have shown that the compounding effects of error propagation in downstream data analysis tasks lead to lower detection accuracy and increased overhead due to unnecessary model updates. Unfortunately, existing work has largely explored anomaly detection and concept drift detection in isolation. We develop AnDri, a system for Anomaly detection in the presence of Drift, which adjusts the normal patterns temporally, and distinguish abnormal subsequences and new concepts. Moreover, it introduces a new clustering method, Adjacent Hierarchical Clustering (AHC), which groups similar subsequences while respecting their temporal locality.

Problem

• Detecting anomalies amidst evolving data distributions (concept drift)
• Distinguishing abnormal changes from normal behavior variations
• Addressing error propagation in downstream analysis tasks

Innovation

• Adaptive anomaly detection with concept drift
• Temporal adjustment of normal patterns
• Adjacent Hierarchical Clustering for subsequences