Adaptive Anomaly Detection in Network Flows With Low-Rank Tensor Decompositions and Deep Unrolling

2024-09-17

IEEE Open Journal of Signal Processing

AI Summary

To address three key challenges in network flow anomaly detection—low data efficiency due to incomplete measurements, poor cross-domain adaptability, and weak interpretability—this paper proposes a novel framework integrating low-rank tensor modeling with deep unfolding. Methodologically, it introduces: (1) a learnable regularization deep-unfolding network driven by a novel block-wise convex approximation algorithm, preserving permutation equivariance with minimal parameters; (2) a Bayesian-inspired, flow-level and time-step-level online adaptive mechanism; and (3) joint training via sparse modeling, homotopy-based optimization, and AUC-approximated gradient descent. Evaluated on synthetic and real-world datasets, the framework reduces required training samples by over 50%, improves average AUC by 8.3%, enables seamless transfer across multiple network topologies, and achieves high accuracy, strong generalization, and intrinsic interpretability.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Computer Vision: Learning & Optimization for CV
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Anomaly detection (AD) is increasingly recognized as a key component for ensuring the resilience of future communication systems. While deep learning has shown state-of-the-art AD performance, its application in critical systems is hindered by concerns regarding training data efficiency, domain adaptation and interpretability. This work considers AD in network flows using incomplete measurements, leveraging a robust tensor decomposition approach and deep unrolling techniques to address these challenges. We first propose a novel block-successive convex approximation algorithm based on a regularized model-fitting objective where the normal flows are modeled as low-rank tensors and anomalies as sparse. An augmentation of the objective is introduced to decrease the computational cost. We apply deep unrolling to derive a novel deep network architecture based on our proposed algorithm, treating the regularization parameters as learnable weights. Inspired by Bayesian approaches, we extend the model architecture to perform online adaptation to per-flow and per-time-step statistics, improving AD performance while maintaining a low parameter count and preserving the problem's permutation equivariances. To optimize the deep network weights for detection performance, we employ a homotopy optimization approach based on an efficient approximation of the area under the receiver operating characteristic curve. Extensive experiments on synthetic and real-world data demonstrate that our proposed deep network architecture exhibits a high training data efficiency, outperforms reference methods, and adapts seamlessly to varying network topologies.

Problem

• Detect anomalies in network flows using incomplete measurements
• Improve training data efficiency and domain adaptation in AD
• Maintain interpretability while leveraging deep learning for AD

Innovation

• Low-rank tensor decomposition for anomaly detection
• Deep unrolling for efficient network architecture
• Online adaptation with Bayesian-inspired techniques