Evaluating Open-Weight LLMs for Generating Structured Threat Information for Autonomous Vehicle Vulnerabilities

2026-07-17

AI Summary

This study addresses the challenge posed by the prevalence of unstructured textual vulnerability reports in autonomous vehicles, which hinders efficient security response and threat analysis. To bridge this gap, the authors introduce CAV-STIXGen, the first dataset mapping Common Vulnerabilities and Exposures (CVEs) in autonomous vehicles to structured threat intelligence formats including STIX, Common Weakness Enumerations (CWE), and MITRE ATT&CK. The work systematically evaluates 11 open-source large language models (LLMs), spanning parameter scales from 4B to 120B, under varying prompting strategies and temperature settings, and further explores a multi-agent LLM architecture for automated threat intelligence generation. Experimental results demonstrate that a single LLM achieves F1 scores of 0.94 and 0.99 on STIX Domain Object (SDO) and CWE mapping tasks, respectively, while the multi-agent configuration attains an SDO F1 score of 0.91, revealing characteristic threat patterns in the domain.

Abstract

Connected and Autonomous Vehicles (CAVs) rely on interconnected software and hardware components, including sensors, Electronic Control Units, in-vehicle infotainment systems, and telematics units, where vulnerabilities can compromise assets, users, and vehicle operations. These vulnerabilities are commonly documented as plain text in the Common Vulnerabilities and Exposures (CVE) database; however, security practitioners require structured information about affected assets, types of weaknesses, and attack behaviors to effectively mitigate the risks from these vulnerabilities. To this end, we evaluate open-weight Large Language Models (LLMs) for generating Structured Threat Information Expression (STIX), a well-known structured format for representing threat information, for CAV-related CVEs. We construct a dataset called CAV-STIXGen that maps CAV vulnerability descriptions to STIX domain objects (SDO), STIX relationship objects (SRO), Common Weakness Enumeration (CWE), and MITRE ATT&CK techniques mappings. Using this dataset, we evaluated 11 open-weight LLMs (4B to 120B parameters) across various prompting strategies and temperatures. Single-model configurations achieve F1 scores of 0.94 for SDO, 0.63 for SRO, and 0.99 for CWE mapping, while complete MITRE ATT&CK mapping remains challenging. In a multi-agent setup, Gemma-4-31B and Codestral-22B achieve F1 scores of 0.91 for SDOs and 0.43 for SROs, respectively. Lastly, we analyze CWE and MITRE ATT&CK co-occurrences to identify recurring threat patterns in the CAV domain, demonstrating how AI-assisted vulnerability-to-STIX translation can automate threat intelligence and prioritize defense in transportation security.

Problem

• Autonomous Vehicles
• Vulnerability
• Structured Threat Information
• CVE
• STIX

Innovation

• Structured Threat Information (STIX)
• Open-weight LLMs
• Autonomous Vehicle Vulnerabilities
• CAV-STIXGen Dataset
• MITRE ATT&CK Mapping