Open World Autoencoding Drift Detection with Novel Class Recognition in Tabular Non-stationary Data Streams

2026-05-28

AI Summary

This work addresses the challenge of simultaneously handling concept drift and emerging novel classes in non-stationary tabular data streams by proposing an unsupervised continual learning approach. The method employs a mirror autoencoder architecture to decouple two complementary tasks: detecting distributional shifts among known classes via reconstruction error and identifying novel classes through density estimation over sample proxy representations. Both components support incremental adaptation, enabling continuous tracking of evolving concepts and reliable discovery of previously unseen categories. To the best of our knowledge, this is the first study to introduce mirror autoencoders to this problem setting. Experimental results demonstrate that the proposed method achieves performance on par with state-of-the-art unsupervised detection and recognition techniques across multiple synthetic data streams.

Abstract

Data stream processing has become a landmark in modern machine learning applications, with concept drifts and novel class appearances posing the primary challenges faced by sophisticated recognition methods. This work proposes an unsupervised concept drift detection method that identifies shifts in known class distributions based on the reconstruction errors of an autoencoder, while also enabling the recognition of novel class samples through density estimation of a proxy representation of samples. Using mirrored autoencoders allows for independent incremental adaptation to changing problem distributions for the two considered tasks, resulting in continuous adjustment to evolving concepts and reliable recognition of unknown samples. Conducted experiments used a diverse set of synthetic tabular data streams, where both concept drifts and the emergence of novelties were observed. The results show that the proposed approach is competitive with current state-of-the-art unsupervised drift detectors and novelty classifiers.

Problem

• concept drift
• novel class recognition
• data streams
• non-stationary data
• tabular data

Innovation

• autoencoder
• concept drift detection
• novel class recognition
• non-stationary data streams
• density estimation