Toward Improved Deep Learning-Based Vulnerability Detection

2024-02-06

International Conference on Software Engineering

AI Summary

Existing deep learning–based vulnerability detectors exhibit significantly degraded performance—averaging over a 35% accuracy drop—when identifying multi-block-unit (MBU) vulnerabilities, which require coordinated triggering across multiple code units. Method: This paper introduces the first formal definition of MBU vulnerabilities and proposes the first empirical evaluation framework specifically designed for cross-unit vulnerability detection. The framework integrates program slicing, consistency-aware annotation, and multi-model comparative experiments. Contribution/Results: Applying this framework to state-of-the-art detectors—including ReVeal, DeepWukong, and LineVul—we systematically expose their detection blind spots on MBU vulnerabilities. Beyond quantifying model-specific omission rates, our work advances the granularity of vulnerability detection from single-code-unit to cross-unit analysis, establishing both a theoretical foundation and a practical toolkit for future model design and evaluation.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Adversarial Learning & Robustness
• Data Mining & Knowledge Management: Anomaly/Outlier Detection

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

Deep learning (DL) has been a common thread across several recent techniques for vulnerability detection. The rise of large, publicly available datasets of vulnerabilities has fueled the learning process underpinning these techniques. While these datasets help the DL-based vulnerability detectors, they also constrain these detectors' predictive abilities. Vulnerabilities in these datasets have to be represented in a certain way, e.g., code lines, functions, or pro-gram slices within which the vulnerabilities exist. We refer to this representation as a base unit. The detectors learn how base units can be vulnerable and then predict whether other base units are vulnerable. We have hypothesized that this focus on individual base units harms the ability of the detectors to properly detect those vul-nerabilities that span multiple base units (or MBU vulnerabilities). For vulnerabilities such as these, a correct detection occurs when all comprising base units are detected as vulnerable. Verifying how existing techniques perform in detecting all parts of a vulnerability is important to establish their effectiveness for other downstream tasks. To evaluate our hypothesis, we conducted a study focusing on three prominent DL-based detectors: ReVeal, DeepWukong, and LineVul. Our study shows that all three detectors contain MBU vulnerabilities in their respective datasets. Further, we observed significant accuracy drops when detecting these types of vulner-abilities. We present our study and a framework that can be used to help DL-based detectors toward the proper inclusion of MBU vulnerabilities.

Problem

• Deep Learning
• Software Vulnerabilities
• Multi-unit Detection

Innovation

• Cross-unit Vulnerability Detection
• Deep Learning Systems
• Accuracy Improvement