Evaluating LLMs for One-Shot Patching of Real and Artificial Vulnerabilities

2025-11-28

AI Summary

Prior evaluations of large language models (LLMs) for vulnerability repair predominantly rely on publicly disclosed vulnerabilities, neglecting systematic assessment on synthetically constructed vulnerabilities—a critical gap in understanding model generalization and robustness. Method: This work introduces the first unified benchmark to systematically evaluate the single-shot automated patching capability of mainstream LLMs—including GPT, LLaMA, DeepSeek, and Mistral—on both real-world and synthetically generated vulnerabilities. Patch correctness is rigorously validated via automated proof-of-vulnerability (PoV) testing. Contribution/Results: Experiments reveal that LLMs achieve significantly higher repair success rates on real vulnerabilities than on synthetic ones; moreover, models exhibit strong complementarity—not mere performance ranking—across vulnerability types and coverage dimensions. Based on these findings, we propose a task-characteristic-driven model selection strategy, providing empirical grounding and methodological guidance for building multi-model collaborative vulnerability repair systems.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Natural Language Processing: (Large) Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Automated vulnerability patching is crucial for software security, and recent advancements in Large Language Models (LLMs) present promising capabilities for automating this task. However, existing research has primarily assessed LLMs using publicly disclosed vulnerabilities, leaving their effectiveness on related artificial vulnerabilities largely unexplored. In this study, we empirically evaluate the patching effectiveness and complementarity of several prominent LLMs, such as OpenAI's GPT variants, LLaMA, DeepSeek, and Mistral models, using both real and artificial vulnerabilities. Our evaluation employs Proof-of-Vulnerability (PoV) test execution to concretely assess whether LLM-generated source code successfully patches vulnerabilities. Our results reveal that LLMs patch real vulnerabilities more effectively compared to artificial ones. Additionally, our analysis reveals significant variability across LLMs in terms of overlapping (multiple LLMs patching the same vulnerabilities) and complementarity (vulnerabilities patched exclusively by a single LLM), emphasizing the importance of selecting appropriate LLMs for effective vulnerability patching.

Problem

• Evaluating LLMs for automated patching of real and artificial software vulnerabilities
• Assessing patching effectiveness using Proof-of-Vulnerability test execution methods
• Analyzing variability and complementarity across different LLM models for vulnerability repair

Innovation

• Evaluating LLMs for vulnerability patching effectiveness
• Using real and artificial vulnerabilities for assessment
• Employing Proof-of-Vulnerability test execution methodology