CyberThreat-Eval: Can Large Language Models Automate Real-World Threat Research?

2026-03-10

AI Summary

This work addresses the lack of systematic evaluation of large language models (LLMs) in automating cybersecurity threat intelligence (CTI) tasks according to the authentic three-stage analyst workflow—triage, deep investigation, and intelligence reporting—and highlights the inadequacy of conventional metrics in capturing operational utility and domain-specific accuracy. To bridge this gap, the authors introduce CyberThreat-Eval, the first enterprise-scale, expert-annotated benchmark encompassing the full CTI workflow, along with analyst-centered evaluation metrics that jointly assess factual accuracy, content quality, and operational cost. By integrating expert annotations, external knowledge bases, and a human feedback-driven iterative refinement mechanism (TRA), the study reveals critical limitations of current LLMs in handling complex threat details and discerning information veracity, while demonstrating that fusing expert and external knowledge is essential for advancing the quality of automated CTI systems.

Technology Category

• Machine Learning: Evaluation and Analysis
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Knowledge Representation and Reasoning: Automated Reasoning and Theorem Proving

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Analyzing Open Source Intelligence (OSINT) from large volumes of data is critical for drafting and publishing comprehensive CTI reports. This process usually follows a three-stage workflow -- triage, deep search and TI drafting. While Large Language Models (LLMs) offer a promising route toward automation, existing benchmarks still have limitations. These benchmarks often consist of tasks that do not reflect real-world analyst workflows. For example, human analysts rarely receive tasks in the form of multiple-choice questions. Also, existing benchmarks often rely on model-centric metrics that emphasize lexical overlap rather than actionable, detailed insights essential for security analysts. Moreover, they typically fail to cover the complete three-stage workflow. To address these issues, we introduce CyberThreat-Eval, which is collected from the daily CTI workflow of a world-leading company. This expert-annotated benchmark assesses LLMs on practical tasks across all three stages as mentioned above. It utilizes analyst-centric metrics that measure factual accuracy, content quality, and operational costs. Our evaluation using this benchmark reveals important insights into the limitations of current LLMs. For example, LLMs often lack the nuanced expertise required to handle complex details and struggle to distinguish between correct and incorrect information. To address these challenges, the CTI workflow incorporates both external ground-truth databases and human expert knowledge. TRA allows human experts to iteratively provide feedback for continuous improvement. The code is available at \href{https://github.com/xschen-beb/CyberThreat-Eval}{\texttt{GitHub}} and \href{https://huggingface.co/datasets/xse/CyberThreat-Eval}{\texttt{HuggingFace}}.

Problem

• Cyber Threat Intelligence
• Large Language Models
• Benchmarking
• OSINT
• Analyst Workflow

Innovation

• CyberThreat-Eval
• threat intelligence automation
• analyst-centric evaluation
• large language models
• OSINT