Real-Time Decorrelation-Based Anomaly Detection for Multivariate Time Series

2025-07-10

AI Summary

This paper addresses real-time anomaly detection in high-dimensional multivariate time series from industrial IoT, requiring single-pass processing, zero historical storage, minimal memory footprint, and ultra-low latency. To this end, we propose Dynamic Decorrelation Detection (DAD), a novel online, sample-by-sample decorrelation modeling framework that departs from conventional distance- or reconstruction-based paradigms. DAD dynamically learns evolving correlation structures via sliding-window correlation matrix estimation, standardized residual monitoring, and an adaptive thresholding mechanism—enabling lightweight, memory-efficient dependency modeling. We further introduce an online hyperparameter tuning strategy tailored for streaming settings. Evaluated on multiple benchmark datasets, DAD achieves state-of-the-art and highly stable detection performance with the lowest memory overhead, notably outperforming existing methods in high-dimensional regimes. Our approach establishes a new benchmark for resource-constrained, real-time anomaly detection in industrial IoT.

Technology Category

• Data Mining & Knowledge Management: Anomaly/Outlier Detection
• Machine Learning: Time-Series/Data Streams
• Application Domains: Internet of Things, Sensor Networks & Smart Cities

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications

Abstract

Anomaly detection (AD) plays a vital role across a wide range of real-world domains by identifying data instances that deviate from expected patterns, potentially signaling critical events such as system failures, fraudulent activities, or rare medical conditions. The demand for real-time AD has surged with the rise of the (Industrial) Internet of Things, where massive volumes of multivariate sensor data must be processed instantaneously. Real-time AD requires methods that not only handle high-dimensional streaming data but also operate in a single-pass manner, without the burden of storing historical instances, thereby ensuring minimal memory usage and fast decision-making. We propose DAD, a novel real-time decorrelation-based anomaly detection method for multivariate time series, based on an online decorrelation learning approach. Unlike traditional proximity-based or reconstruction-based detectors that process entire data or windowed instances, DAD dynamically learns and monitors the correlation structure of data sample by sample in a single pass, enabling efficient and effective detection. To support more realistic benchmarking practices, we also introduce a practical hyperparameter tuning strategy tailored for real-time anomaly detection scenarios. Extensive experiments on widely used benchmark datasets demonstrate that DAD achieves the most consistent and superior performance across diverse anomaly types compared to state-of-the-art methods. Crucially, its robustness to increasing dimensionality makes it particularly well-suited for real-time, high-dimensional data streams. Ultimately, DAD not only strikes an optimal balance between detection efficacy and computational efficiency but also sets a new standard for real-time, memory-constrained anomaly detection.

Problem

• Detects anomalies in real-time multivariate time series data
• Handles high-dimensional streaming data with minimal memory usage
• Improves robustness and efficiency in anomaly detection methods

Innovation

• Online decorrelation learning for anomaly detection
• Single-pass processing of high-dimensional streaming data
• Hyperparameter tuning for real-time scenarios