GenTI: Benchmarking LLMs for Autonomous IDPS Rule Generation for Unseen Attacks

2026-06-04

AI Summary

This study addresses the limitations of traditional rule-based intrusion detection and prevention systems (IDPS), which struggle to detect novel and zero-day attacks and lack structured data to support automated rule generation. To overcome these challenges, the authors introduce the first large-scale GTI dataset and propose an automated rule generation framework that integrates rule-level cyber threat intelligence with large language models (LLMs). The framework leverages structured prompt engineering, chain-of-thought (CoT) reasoning, and a chain-of-verification (CoVe) loop to automatically translate analyst inputs and malicious payloads into deployable Snort/Suricata rules. Experimental results demonstrate that the generated rules achieve a composite quality score of 89.4%, cover 94.8% of relevant threat intelligence, improve detection rates for previously unseen attacks from 45% to 87.4%, and reduce false positive rates from 8.5% to 2.3%.

Abstract

Rule-based Intrusion Detection and Prevention Systems (IDPS) offer precise attack detection as well as mitigation, however their manually crafted, signature-driven rules limit adaptability to emerging and zero-day threats. Additionally, existing public datasets (e.g., CICIDS2017, UNSW-NB15) focus on traffic classification and provide little structured information to support automatic rule synthesis or prevention logic. To address this gap, we propose Generative Thread Intelligence (GenTI) \footnote{GenTI refers to the proposed framework, and GTI refers to the dataset.} an LLM-driven benchmark for automatic generation of IDPS rules targeting unseen attacks. The dataset (GTI) aggregates over 150k detection and prevention rules from Snort, Suricata, Emerging Threats, as well as 50k YARA, each annotated with protocol behavior, payload signatures, contextual relationships, mappings to Cyber Threat Intelligence (CTI), along with actionable response types (alert, drop, reject). Moreover, on top of this corpus we design an LLM-based pipeline that transforms analyst prompts and representative payloads into deployable rules via structured prompt engineering, Chain-of-Thought (CoT) reasoning, as well as a Chain-of-Verification (CoVe) loop for syntactic, semantic, and security validation. The generated rules are executed in real time on (Snort/Suricata) and evaluated by syntax accuracy, semantic similarity, CTI coverage, security effectiveness as well as unseen attacks detection. Furthermore, our GenTI instantiation achieves a composite rule-quality score of 89.4\%, with 94.8\% CTI coverage, improving unseen attacks detection from 45\% to 87.4\% and reducing the false-positive rate from 8.5\% to 2.3\%. Overall, GenTI establishes the first large-scale benchmark that tightly couples rule-level CTI with LLM-based automation, enabling adaptive, self-evolving IDPS.

Problem

• IDPS
• zero-day attacks
• rule generation
• Cyber Threat Intelligence
• unseen attacks

Innovation

• LLM-driven IDPS
• rule generation
• Chain-of-Verification
• Cyber Threat Intelligence
• zero-day attack detection