Adaptive Intrusion Detection System using Transformer-Based Neural Networks and Continual Learning Approach with Adversarial Investigation

2026-08-05

AI Summary

This work addresses the performance degradation of network intrusion detection systems after deployment due to threat distribution shifts and the limitations of existing continual learning approaches, which suffer from catastrophic forgetting, high computational overhead, and insufficient consideration of security risks associated with replay buffers. The authors propose a Class-Instance Incremental (CII) learning scenario that integrates a Tabular Transformer encoder with a class-balanced experience replay mechanism, consistently replaying benign traffic during each update to stabilize the decision boundary while systematically evaluating the replay buffer as a potential attack surface. Experiments on CICIDS2017 show that the model achieves accuracies of 0.9994 and 0.9989 under conventional class-incremental and CII settings, respectively, significantly outperforming baselines. However, it is highly vulnerable to data poisoning: a mere 1% label-flipping attack reduces accuracy to 0.0053, and backdoor attacks achieve a 95% success rate, exposing critical security weaknesses in current continual learning paradigms for safety-critical applications.

Abstract

Network intrusion detection systems (IDS) trained on fixed traffic snapshots decay silently after deployment as threat distributions shift. Fine-tuning models on new attacks triggers catastrophic forgetting, while retraining from scratch is computationally infeasible. Replay-based continual learning counters this, but existing methods unrealistically confine benign traffic to a single early task and ignore the replay buffer as a potential attack surface. To address this, we present an adaptive IDS framework coupling a tabular transformer encoder with a class balanced experience replay buffer that replays benign traffic at every update to stabilize decision boundaries. We introduce the class-instance incremental (CII) scenario where benign flows reappear alongside new attacks as a more faithful stress test, and probe the buffer with overt label flipping and stealthy backdoor poisoning attacks. On the CICIDS2017 benchmark, our framework achieved 0.9994 accuracy under the traditional class incremental setup and 0.9989 under CII, with negligible forgetting, drastically outperforming sequential fine-tuning (0.0052), EWC (0.0324), LwF (0.0699), and iCaRL (0.8770) baselines. While injecting benign traffic into every experience proves essential for preventing forgetting, the replay buffer introduces critical vulnerabilities. Label-flipping collapses the model entirely (0.0053 accuracy at a 1% budget), and the backdoor maintains 0.97 overall accuracy while driving the attack success rate on trigger flows to 95%, evading standard monitoring. Ultimately, while a modest replay budget recovers near-joint-training performance, ensuring buffer integrity emerges as a strict operational requirement.

Problem

• intrusion detection
• catastrophic forgetting
• continual learning
• replay buffer
• adversarial attack

Innovation

• Continual Learning
• Transformer-based IDS
• Experience Replay
• Class-Instance Incremental Learning
• Adversarial Buffer Attacks