Improving Data Curation of Software Vulnerability Patches through Uncertainty Quantification

2024-11-18

arXiv.org

AI Summary

Open-source vulnerability patch datasets suffer from inaccurate labeling and critical sample omissions, severely degrading the performance of downstream security analysis models. To address this, we propose an uncertainty quantification (UQ)-guided, utility-oriented data cleaning framework—the first to integrate model ensembling and heteroscedastic uncertainty modeling for patch quality assessment. Our method combines Monte Carlo Dropout with confidence estimation to automatically identify high-utility patches and filter low-quality samples. This establishes a novel UQ-driven data purification paradigm that preserves dataset representativeness while significantly improving vulnerability prediction accuracy, reducing training time, and lowering computational energy consumption. Extensive experiments demonstrate the effectiveness, scalability, and practical utility of UQ-informed patch selection across diverse vulnerability detection tasks.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Computer Vision: Adversarial Attacks & Robustness
• Reasoning under Uncertainty: Uncertainty Representations

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Web Mining and Content Analysis: Web data integration and cleaning
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

The changesets (or patches) that fix open source software vulnerabilities form critical datasets for various machine learning security-enhancing applications, such as automated vulnerability patching and silent fix detection. These patch datasets are derived from extensive collections of historical vulnerability fixes, maintained in databases like the Common Vulnerabilities and Exposures list and the National Vulnerability Database. However, since these databases focus on rapid notification to the security community, they contain significant inaccuracies and omissions that have a negative impact on downstream software security quality assurance tasks. In this paper, we propose an approach employing Uncertainty Quantification (UQ) to curate datasets of publicly-available software vulnerability patches. Our methodology leverages machine learning models that incorporate UQ to differentiate between patches based on their potential utility. We begin by evaluating a number of popular UQ techniques, including Vanilla, Monte Carlo Dropout, and Model Ensemble, as well as homoscedastic and heteroscedastic models of noise. Our findings indicate that Model Ensemble and heteroscedastic models are the best choices for vulnerability patch datasets. Based on these UQ modeling choices, we propose a heuristic that uses UQ to filter out lower quality instances and select instances with high utility value from the vulnerability dataset. Using our approach, we observe an improvement in predictive performance and significant reduction of model training time (i.e., energy consumption) for a state-of-the-art vulnerability prediction model.

Problem

• Addressing inaccuracies in software vulnerability patch datasets
• Improving data quality for machine learning security applications
• Reducing negative impact on software security quality assurance

Innovation

• Uses Uncertainty Quantification to curate vulnerability patch datasets
• Leverages Model Ensemble and heteroscedastic models for UQ
• Proposes a UQ-based heuristic to filter low-quality patches