On the Validity of Traditional Vulnerability Scoring Systems for Adversarial Attacks against LLMs

2024-12-28

AI Summary

Traditional vulnerability scoring systems (e.g., CVSS, EPSS) exhibit fundamental limitations in assessing adversarial attacks (AAs) against large language models (LLMs), as their static, context-agnostic design fails to capture semantic-level threat distinctions—particularly for prompt injection and other input-manipulation attacks. Method: We systematically evaluate 56 adversarial attack variants across GPT-4, Claude, and Llama, quantifying score dispersion via the coefficient of variation (CV) and benchmarking against multi-source adversarial datasets. Contribution/Results: Empirical analysis reveals that existing metrics assign uniformly low, highly convergent scores to most attacks (CV < 0.08), demonstrating severe discriminative inadequacy. To address this, we propose the first flexible, context-aware scoring framework specifically designed for generative AI—grounded in adaptive risk modeling, dynamic contextualization, and semantic impact quantification. This work establishes a novel paradigm for rigorous, actionable LLM security assessment.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Adversarial Attacks & Robustness
• Multiagent Systems: Adversarial Agents

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Security and Privacy: Large-scale security measurements
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

This research investigates the effectiveness of established vulnerability metrics, such as the Common Vulnerability Scoring System (CVSS), in evaluating attacks against Large Language Models (LLMs), with a focus on Adversarial Attacks (AAs). The study explores the influence of both general and specific metric factors in determining vulnerability scores, providing new perspectives on potential enhancements to these metrics. This study adopts a quantitative approach, calculating and comparing the coefficient of variation of vulnerability scores across 56 adversarial attacks on LLMs. The attacks, sourced from various research papers, and obtained through online databases, were evaluated using multiple vulnerability metrics. Scores were determined by averaging the values assessed by three distinct LLMs. The results indicate that existing scoring-systems yield vulnerability scores with minimal variation across different attacks, suggesting that many of the metric factors are inadequate for assessing adversarial attacks on LLMs. This is particularly true for context-specific factors or those with predefined value sets, such as those in CVSS. These findings support the hypothesis that current vulnerability metrics, especially those with rigid values, are limited in evaluating AAs on LLMs, highlighting the need for the development of more flexible, generalized metrics tailored to such attacks. This research offers a fresh analysis of the effectiveness and applicability of established vulnerability metrics, particularly in the context of Adversarial Attacks on Large Language Models, both of which have gained significant attention in recent years. Through extensive testing and calculations, the study underscores the limitations of these metrics and opens up new avenues for improving and refining vulnerability assessment frameworks specifically tailored for LLMs.

Problem

• Vulnerability Assessment
• Large Language Models
• Adversarial Attacks

Innovation

• Adversarial Attacks
• Language Models
• Vulnerability Assessment