Vul4Py: Benchmarking Automated Vulnerability Repair in Python with Paired Exploit and Functional Oracles

2026-08-01

AI Summary

This work addresses the lack of verifiable benchmarks in Python automated vulnerability repair (AVR), where existing approaches often overlook functional regressions introduced by patches due to reliance solely on exploit-based or functional tests. To bridge this gap, we propose Vul4Py—the first Python AVR benchmark incorporating a paired-oracle mechanism that provides dual validation through both exploits and native functional tests for 100 real-world vulnerabilities spanning 60 distinct CWEs, all within reproducible isolated environments. Using this benchmark, we conduct a unified evaluation of three repair paradigms: specialized AVR tools, direct prompting of large language models (LLMs), and software engineering agents. Our experiments show that the OpenHands agent successfully repairs 41 vulnerabilities, substantially outperforming LLM prompting (4 fixes) and specialized tools (2 fixes), with 98% of its valid patches being semantically equivalent to developer-authored patches.

Abstract

Automated Vulnerability Repair (AVR) has advanced rapidly across program analysis, machine learning, and Large Language Models (LLMs), but a verifiable, head-to-head comparison of AVR approaches on Python is still missing. Python underpins critical web, data, and machine-learning infrastructure, yet existing Python benchmarks accept a patch on the strength of a proof-of-concept exploit alone, or apply a functional test only on the subset of entries whose upstream project happens to ship one. Both therefore miss functional regressions, in which a patch defeats the exploit but breaks unrelated behavior. We present Vul4Py, a Python AVR benchmark in which every entry carries a paired oracle: an exploit oracle that must fail on the vulnerable revision and pass on the fixed one, together with a project-native pytest functional oracle that must pass on both. Vul4Py comprises 100 real vulnerabilities from 60 open-source projects, spanning 60 distinct CWEs and the years 2017 to 2025, each packaged with a pinned, reproducible per-instance environment. Using Vul4Py, we compare six approaches in three categories: a specialized vulnerability repair tool, directly prompted LLMs, and software engineering agents. The agents dominate: OpenHands repairs 41 of 100 vulnerabilities, against 4 for the strongest directly prompted LLM and 2 for the specialized tool, despite all three sharing the same backbone model. The paired oracle is what makes these counts trustworthy: it rejects 15 of the 119 patches that an exploit-only oracle would accept, and 98 of the 104 patches it admits are manually confirmed to be semantically equivalent to the developer's patches

Problem

• Automated Vulnerability Repair
• Python
• Functional Regression
• Benchmark
• Exploit Oracle

Innovation

• Automated Vulnerability Repair
• Paired Oracle
• Functional Regression
• Python Benchmark
• Software Engineering Agents