Adaptive NAD: Online and Self-adaptive Unsupervised Network Anomaly Detector

2024-10-30

AI Summary

Traditional offline unsupervised anomaly detection systems suffer from poor adaptability and weak interpretability in IoT environments, where attacks dynamically evolve. To address this, we propose an online-updatable, label-free, and interpretable unsupervised anomaly detection framework. Our key contributions are: (1) an interpretable two-tier detection strategy that automatically generates high-confidence pseudo-labels; and (2) an online learning mechanism based on dynamic threshold computation, enabling the model to self-adapt to previously unseen attacks. Evaluated on CIC-Darknet2020, CIC-DoHBrw-2020, and Edge-IIoTset datasets, our method achieves SPAUC improvements of 5.4%, 23.0%, and 3.2%, respectively—outperforming state-of-the-art approaches. The implementation is publicly available.

Technology Category

• Data Mining & Knowledge Management: Anomaly/Outlier Detection
• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Unsupervised & Self-Supervised Learning

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

The widespread usage of the Internet of Things (IoT) has raised the risks of cyber threats, thus developing Anomaly Detection Systems (ADSs) that can adapt to evolving or new attacks is critical. Previous studies primarily focused on offline unsupervised learning methods to safeguard ADSs, which is not applicable in practical real-world applications. Besides, most of them strongly rely on assumptions of known legitimates and fail to satisfy the interpretable requirements in security applications, creating barriers to the adoption in practice. In this paper, we design Adaptive NAD, a general framework to improve and interpret online unsupervised anomaly detection in security domains. An interpretable two-layer anomaly detection strategy is proposed to generate reliable high-confidence pseudo-labels. Then, an online learning scheme is introduced to update Adaptive NAD by a novel threshold calculation technique to adapt to new threats. Experimental results demonstrate that Adaptive NAD achieves more than 5.4%, 23.0%, and 3.2% improvements in SPAUC compared with state-of-the-art solutions on the CIC-Darknet2020, CIC-DoHBrw-2020, and Edge-IIoTset datasets, respectively. The code is released at https://github.com/MyLearnCodeSpace/Adaptive-NAD.

Problem

• Improve online unsupervised anomaly detection
• Adapt to new evolving cyber threats
• Generate interpretable high-confidence pseudo-labels

Innovation

• Online unsupervised anomaly detection
• Interpretable two-layer detection strategy
• Novel threshold calculation technique