Technology Category
Computer Vision: Adversarial Attacks & RobustnessKnowledge Representation and Reasoning: Qualitative ReasoningPlanning, Routing, and Scheduling: Replanning and Plan Repair
R+R: Security Vulnerability Dataset Quality Is Critical
🤖 AI Summary
This paper identifies severe data quality issues in mainstream datasets for vulnerability detection and repair—namely, high duplication rates, pervasive label errors (56%), and incomplete samples (44%)—leading to substantial overestimation of model performance (e.g., VulRepair’s true accuracy drops from the reported 44% to 9–13%). Method: The authors introduce the first systematic quantification of these three quality dimensions and propose a novel paradigm integrating automated deduplication, human-audited label verification, and CWE-guided completeness assessment, alongside constructing a high-quality pretraining corpus. Contribution/Results: Only 31% of samples meet the proposed high-quality standard; leveraging deduplicated bug-fix corpora via transfer learning significantly improves models’ real-world performance. This work establishes the first benchmark for data quality evaluation in vulnerability research and provides a concrete, actionable pathway for dataset curation and model validation.
Technology Category
Application Category
📝 Abstract
Large Language Models (LLMs) are of great interest in vulnerability detection and repair. The effectiveness of these models hinges on the quality of the datasets used for both training and evaluation. Our investigation reveals that a number of studies featured in prominent software engineering conferences have employed datasets that are plagued by high duplication rates, questionable label accuracy, and incomplete samples. Using these datasets for experimentation will yield incorrect results that are significantly different from actual expected behavior. For example, the state-of-the-art VulRepair Model, which is reported to have 44% accuracy, on average yielded 9% accuracy when test-set duplicates were removed from its training set and 13% accuracy when training-set duplicates were removed from its test set. In an effort to tackle these data quality concerns, we have retrained models from several papers without duplicates and conducted an accuracy assessment of labels for the top ten most hazardous Common Weakness Enumerations (CWEs). Our findings indicate that 56% of the samples had incorrect labels and 44% comprised incomplete samples--only 31% were both accurate and complete. Finally, we employ transfer learning using a large deduplicated bugfix corpus to show that these models can exhibit better performance if given larger amounts of high-quality pre-training data, leading us to conclude that while previous studies have over-estimated performance due to poor dataset quality, this does not demonstrate that better performance is not possible.
Problem
Research questions and friction points this paper is trying to address.
High duplication rates in vulnerability datasets affect model accuracy.
Incorrect and incomplete labels in datasets lead to unreliable results.
Improved dataset quality enhances Large Language Models' vulnerability detection performance.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Retrained models using deduplicated datasets
Assessed label accuracy for top CWEs
Employed transfer learning with high-quality data
🔎 Similar Papers
No similar papers found.
