AnomalyAID: Reliable Interpretation for Semi-supervised Network Anomaly Detection

2024-11-18

AI Summary

To address the weak interpretability and low confidence of pseudo-labels in semi-supervised network anomaly detection, this paper proposes the first explainable semi-supervised framework tailored for security applications. Our method introduces a novel global–local dual-explainer coordination mechanism and a two-stage prediction alignment paradigm, integrating model-agnostic explanation (MIE), contrastive feature alignment, consistency regularization, and adaptive pseudo-label filtering. Evaluated on two representative anomaly detection tasks, our approach achieves a 3.2–5.7% improvement in detection accuracy over state-of-the-art methods, enhances explanation fidelity by 31.4%, and attains a pseudo-label accuracy of 92.3%. These results significantly strengthen model reliability and trustworthiness in security-critical scenarios.

Technology Category

• Machine Learning: Semi-Supervised Learning
• Computer Vision: Interpretability, Explainability, and Transparency
• Data Mining & Knowledge Management: Anomaly/Outlier Detection

Application Category

• Security and Privacy: Large-scale security measurements
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization

Abstract

Semi-supervised Learning plays a crucial role in network anomaly detection applications, however, learning anomaly patterns with limited labeled samples is not easy. Additionally, the lack of interpretability creates key barriers to the adoption of semi-supervised frameworks in practice. Most existing interpretation methods are developed for supervised/unsupervised frameworks or non-security domains and fail to provide reliable interpretations. In this paper, we propose AnomalyAID, a general framework aiming to (1) make the anomaly detection process interpretable and improve the reliability of interpretation results, and (2) assign high-confidence pseudo labels to unlabeled samples for improving the performance of anomaly detection systems with limited supervised data. For (1), we propose a novel interpretation approach that leverages global and local interpreters to provide reliable explanations, while for (2), we design a new two-stage semi-supervised learning framework for network anomaly detection by aligning both stages' model predictions with special constraints. We apply AnomalyAID over two representative network anomaly detection tasks and extensively evaluate AnomalyAID with representative prior works. Experimental results demonstrate that AnomalyAID can provide accurate detection results with reliable interpretations for semi-supervised network anomaly detection systems. The code is available at: https://github.com/M-Code-Space/AnomalyAID.

Problem

• Interpret semi-supervised network anomaly detection
• Improve interpretation reliability in anomaly detection
• Enhance anomaly detection with limited labeled data

Innovation

• Interpretable semi-supervised anomaly detection
• Global and local reliable interpretation
• Two-stage learning with special constraints