A Hybrid Framework for Real-Time Data Drift and Anomaly Identification Using Hierarchical Temporal Memory and Statistical Tests

2025-04-24

International journal of mathematical, engineering and management sciences

AI Summary

This paper addresses the challenge of real-time joint detection of data drift and anomalies in machine learning. Methodologically, it proposes an online streaming framework integrating Hierarchical Temporal Memory (HTM) with Sequential Probability Ratio Test (SPRT). It introduces, for the first time, a synergistic HTM–SPRT mechanism that overcomes HTM’s inherent univariate limitation, via a multi-column HTM ensemble coupled with a lightweight neural fusion network—establishing the first HTM-enhanced paradigm supporting multivariate supervised anomaly detection. Experiments on real-world telecom datasets demonstrate that the method significantly outperforms classical drift detectors—including Kolmogorov–Smirnov test, Wasserstein distance, and Population Stability Index—in detection accuracy, dynamic adaptability, and false positive rate reduction, while incurring lower computational overhead and eliminating the need for periodic retraining.

Technology Category

• Machine Learning: Time-Series/Data Streams
• Data Mining & Knowledge Management: Anomaly/Outlier Detection
• Humans and AI: Human-in-the-loop Machine Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Data Drift refers to the phenomenon where the generating model behind the data changes over time. Due to data drift, any model built on the past training data becomes less relevant and inaccurate over time. Thus, detecting and controlling for data drift is critical in machine learning models. Hierarchical Temporal Memory (HTM) is a machine learning model developed by Jeff Hawkins, inspired by how the human brain processes information. It is a biologically inspired model of memory similar in structure to the neocortex and whose performance is claimed to be comparable to state of the art models in detecting anomalies in time series data. Another unique benefit of HTMs is their independence from training and testing cycles; all the learning takes place online with streaming data, and no separate training and testing cycle is required. In the sequential learning paradigm, the Sequential Probability Ratio Test (SPRT) offers unique benefits for online learning and inference. This paper proposes a novel hybrid framework combining HTM and SPRT for real-time data drift detection and anomaly identification. Unlike existing data drift methods, our approach eliminates frequent retraining and ensures low false positive rates. HTMs currently work with one dimensional or univariate data. In a second study, we also propose an application of HTM in a multidimensional supervised scenario for anomaly detection by combining the outputs of multiple HTM columns, one for each data dimension, through a neural network. Experimental evaluations demonstrate that the proposed method outperforms conventional drift detection techniques like the Kolmogorov-Smirnov (KS) test, Wasserstein distance, and Population Stability Index (PSI) in terms of accuracy, adaptability, and computational efficiency. Our experiments also provide insights into optimizing hyperparameters for real-time deployment in domains such as Telecom.

Problem

• Detect real-time data drift in machine learning models
• Identify anomalies using HTM and SPRT hybrid framework
• Improve accuracy and efficiency in drift detection methods

Innovation

• Hybrid framework combining HTM and SPRT
• Real-time data drift and anomaly detection
• Multidimensional HTM with neural network integration