Prompting the Priorities: A First Look at Evaluating LLMs for Vulnerability Triage and Prioritization

2025-10-21

AI Summary

Security analysts face mounting challenges in efficiently processing massive, heterogeneous vulnerability data, while the potential of large language models (LLMs) for automated vulnerability semantic parsing and prioritization remains empirically underexplored. This work presents the first systematic evaluation of four LLMs—ChatGPT, Claude, Gemini, and DeepSeek—on their ability to predict four SSVC (Stakeholder-Specific Vulnerability Categorization) decision points: Exploitation, Automatable, Technical Impact, and Mission and Wellbeing. Leveraging the VulZoo dataset and 12 prompt engineering strategies—including zero-shot, few-shot, and chain-of-thought—we conduct over 165,000 inference queries. Results show Gemini with exemplar-based prompting achieves top performance on three decision points; DeepSeek attains fair-level agreement under weighted consistency metrics; overall model effectiveness is moderate but exhibits a systematic bias toward overestimating risk severity. The study establishes a reproducible methodology and empirical benchmark for LLM-augmented vulnerability response.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Security analysts face increasing pressure to triage large and complex vulnerability backlogs. Large Language Models (LLMs) offer a potential aid by automating parts of the interpretation process. We evaluate four models (ChatGPT, Claude, Gemini, and DeepSeek) across twelve prompting techniques to interpret semi-structured and unstructured vulnerability information. As a concrete use case, we test each model's ability to predict decision points in the Stakeholder-Specific Vulnerability Categorization (SSVC) framework: Exploitation, Automatable, Technical Impact, and Mission and Wellbeing. Using 384 real-world vulnerabilities from the VulZoo dataset, we issued more than 165,000 queries to assess performance under prompting styles including one-shot, few-shot, and chain-of-thought. We report F1 scores for each SSVC decision point and Cohen's kappa (weighted and unweighted) for the final SSVC decision outcomes. Gemini consistently ranked highest, leading on three of four decision points and yielding the most correct recommendations. Prompting with exemplars generally improved accuracy, although all models struggled on some decision points. Only DeepSeek achieved fair agreement under weighted metrics, and all models tended to over-predict risk. Overall, current LLMs do not replace expert judgment. However, specific LLM and prompt combinations show moderate effectiveness for targeted SSVC decisions. When applied with care, LLMs can support vulnerability prioritization workflows and help security teams respond more efficiently to emerging threats.

Problem

• Evaluating LLMs for automating vulnerability triage and prioritization processes
• Testing models' ability to predict SSVC framework decision points accurately
• Assessing LLM performance in interpreting semi-structured vulnerability information

Innovation

• Evaluated four LLMs using twelve prompting techniques
• Tested models on predicting SSVC framework decision points
• Applied one-shot few-shot and chain-of-thought prompting styles