A Structured Cyber Threat Intelligence Dataset Using STIX 2.1 Entities and MITRE ATT&CK Mappings

2026-07-25

AI Summary

This study addresses the challenges of extracting critical entities and attack behaviors from unstructured cyber threat intelligence (CTI) reports and the absence of standardized datasets that preserve complex relationships while aligning with the MITRE ATT&CK framework. To bridge this gap, the authors construct a manually annotated dataset comprising 150 English-language CTI reports, modeling entities and relationships using the STIX 2.1 graph structure and mapping them with fine-grained precision to MITRE ATT&CK techniques. This yields the first human-validated, gold-standard benchmark aligned between STIX and MITRE ATT&CK, encompassing 4,777 entities, 5,817 relationships, and 1,273 attack patterns. Experimental evaluation demonstrates that locally deployed large language models (e.g., Qwen-3.6B/27B) achieve a Cohen’s Kappa of 0.803, micro-F1 scores exceeding 92%, and a false positive rate below 5% in annotation consistency assessments, confirming the feasibility of leveraging large language models to support human review.

Abstract

Cyber threat intelligence (CTI) reports are typically written in unstructured formats, which complicates the extraction and analysis of important entities and adversarial behaviors. Although existing CTI research provides extraction tools, knowledge-graph frameworks, and MITRE ATT&CK mapped datasets, curated report-level datasets that preserve complex entity relationships and normalized adversarial behaviors remain limited. To address this limitation, this study presents a manually constructed dataset of 150 English-language CTI reports, each represented as STIX 2.1 based graphs, which includes 4,777 STIX entities, 5,817 STIX relationships in total, and 1,273 STIX attack-pattern entities (adversarial behaviors) mapped to 269 unique MITRE ATT&CK Enterprise techniques and sub-techniques. Twenty five randomly sampled reports were independently assessed by two cybersecurity researchers, which shows substantial inter-rater agreement. Disagreements were subsequently adjudicated to establish a gold-standard reference dataset. Four locally deployed open-source LLMs were evaluated as automated judges against this adjudicated reference sample. Qwen3.6:27B achieved the strongest overall performance, with a maximum kappa score of 0.803, micro-F1 scores exceeding 92%, and false-positive rates below 5%. The dataset provides a benchmark for CTI information extraction, knowledge-graph construction, incident analysis, and threat attribution. The findings further indicate that locally deployed LLMs can support human reviewers in identifying annotation inconsistencies, but expert validation remains essential.

Problem

• Cyber Threat Intelligence
• STIX
• MITRE ATT&CK
• structured dataset
• entity relationship

Innovation

• STIX 2.1
• MITRE ATT&CK
• Cyber Threat Intelligence
• Knowledge Graph
• Large Language Models