A Hierarchical Security Events Correlation Model for Real-time Cyber Threat Detection and Response

2023-12-02

arXiv.org

AI Summary

To address the redundancy and delayed response caused by intrusion detection system (IDS) alert flooding, this paper proposes a real-time hierarchical correlation method operating directly on raw network events—bypassing conventional alert-level correlation. It introduces the first event-level hierarchical modeling and correlation framework, integrating similarity-based matching with graph-structured analysis. The method employs lightweight event clustering and semantic association algorithms to perform pre-detection integration of heterogeneous network events. Evaluated on the DARPA99 dataset, it compresses raw events by 87%, generating approximately 21,000 semantically cohesive clusters within 30 seconds. This substantially reduces alert volume and analytical overhead, while significantly improving both the timeliness and interpretability of threat perception.

Abstract

Intrusion detection systems perform post-compromise detection of security breaches whenever preventive measures such as firewalls do not avert an attack. However, these systems raise a vast number of alerts that must be analysed and triaged by security analysts. This process is largely manual, tedious and time-consuming. Alert correlation is a technique that tries to reduce the number of intrusion alerts by aggregating those that are related in some way. However, the correlation is performed outside the IDS through third-party systems and tools, after the high volume of alerts has already been raised. These other third-party systems add to the complexity of security operations. In this paper, we build on the very researched area of correlation techniques by developing a novel hierarchical event correlation model that promises to reduce the number of alerts issued by an Intrusion Detection System. This is achieved by correlating the events before the IDS classifies them. The proposed model takes the best of features from similarity and graph-based correlation techniques to deliver an ensemble capability not possible by either approach separately. Further, we propose a correlation process for correlation of events rather than alerts as is the case in current art. We further develop our own correlation and clustering algorithm which is tailor-made to the correlation and clustering of network event data. The model is implemented as a proof of concept with experiments run on the DARPA 99 Intrusion detection set. The correlation achieved 87 percent data reduction through aggregation, producing nearly 21000 clusters in about 30 seconds.

Problem

• Alert Correlation
• Cybersecurity
• Network Monitoring

Innovation

• Alert Correlation
• Real-time Processing
• Efficiency Improvement