State Frequency Estimation for Anomaly Detection

2024-12-04

arXiv.org

AI Summary

Existing state-machine-based unsupervised network flow anomaly detection methods employ static anomaly scoring, rendering them ineffective against attacker-crafted near-normal behavioral trajectories and resulting in high false negatives. To address this, we propose SEQUENT—a dynamic, adaptive anomaly detection framework that pioneers the incorporation of state visit frequency into finite-state machine modeling. During inference, SEQUENT dynamically adjusts anomaly scores in real time based on observed traffic patterns, and employs root-cause graph reasoning to generate interpretable, actionable alerts—enabling alert aggregation and forensic analysis. The method integrates three core components: state-frequency statistics, dynamic weighted scoring, and graph-based causal inference. Extensive evaluation on three NetFlow datasets demonstrates that SEQUENT significantly improves detection rates while reducing false positives, achieving both strong adaptability to evolving traffic and high interpretability—two critical yet often conflicting objectives in operational network security.

Technology Category

• Reasoning under Uncertainty: Sequential Decision Making
• Data Mining & Knowledge Management: Anomaly/Outlier Detection
• Machine Learning: Time-Series/Data Streams

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Security and Privacy: Large-scale security measurements

Abstract

Many works have studied the efficacy of state machines for detecting anomalies within NetFlows. These works typically learn a model from unlabeled data and compute anomaly scores for arbitrary traces based on their likelihood of occurrence or how well they fit within the model. However, these methods do not dynamically adapt their scores based on the traces seen at test time. This becomes a problem when an adversary produces seemingly common traces in their attack, causing the model to miss the detection by assigning low anomaly scores. We propose SEQUENT, a new approach that uses the state visit frequency to adapt its scoring for anomaly detection dynamically. SEQUENT subsequently uses the scores to generate root causes for anomalies. These allow the grouping of alarms and simplify the analysis of anomalies. Our evaluation of SEQUENT on three NetFlow datasets indicates that our approach outperforms existing methods, demonstrating its effectiveness in detecting anomalies.

Problem

• Dynamic anomaly score adaptation in state machines
• Detection of adversarial traces in network anomalies
• Root cause analysis and alarm grouping for anomalies

Innovation

• Dynamic scoring using state visit frequency
• Unsupervised anomaly detection with SEQUENT
• Root cause generation for grouped alarms