Today's Cat Is Tomorrow's Dog: Accounting for Time-Based Changes in the Labels of ML Vulnerability Detection Approaches

2025-06-13

AI Summary

This paper addresses the training-deployment label inconsistency problem in machine learning–based vulnerability detection, arising from the temporal evolution of vulnerability labels. We propose a temporally consistent data reconstruction method that strictly constrains both training and test labels to depend solely on information available at the corresponding timepoints, thereby eliminating retrospective bias. To our knowledge, this is the first work to systematically formalize the time-sensitivity of vulnerability labels and to introduce the Mann-Kendall trend test as a metric for quantifying model temporal learning capability. Empirical evaluation across four temporal vulnerability datasets (BigVul subsets augmented with NVD) and five state-of-the-art models—Code2Vec, CodeBERT, LineVul, ReGVD, and VulDeePecker—reveals no statistically significant year-over-year performance improvement. These results refute the implicit assumption that “more historical data necessarily improves detection performance” and expose a pervasive temporal misalignment issue in current vulnerability detection methodologies.

Technology Category

• Data Mining & Knowledge Management: Anomaly/Outlier Detection
• Machine Learning: Calibration & Uncertainty Quantification
• Computer Vision: Image and Video Retrieval

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Web data provenance, reliability, and authenticity
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Vulnerability datasets used for ML testing implicitly contain retrospective information. When tested on the field, one can only use the labels available at the time of training and testing (e.g. seen and assumed negatives). As vulnerabilities are discovered across calendar time, labels change and past performance is not necessarily aligned with future performance. Past works only considered the slices of the whole history (e.g. DiverseVUl) or individual differences between releases (e.g. Jimenez et al. ESEC/FSE 2019). Such approaches are either too optimistic in training (e.g. the whole history) or too conservative (e.g. consecutive releases). We propose a method to restructure a dataset into a series of datasets in which both training and testing labels change to account for the knowledge available at the time. If the model is actually learning, it should improve its performance over time as more data becomes available and data becomes more stable, an effect that can be checked with the Mann-Kendall test. We validate our methodology for vulnerability detection with 4 time-based datasets (3 projects from BigVul dataset + Vuldeepecker's NVD) and 5 ML models (Code2Vec, CodeBERT, LineVul, ReGVD, and Vuldeepecker). In contrast to the intuitive expectation (more retrospective information, better performance), the trend results show that performance changes inconsistently across the years, showing that most models are not learning.

Problem

• Addresses time-based label changes in ML vulnerability detection datasets
• Proposes restructuring datasets to reflect historical label knowledge
• Validates methodology showing inconsistent model performance over time

Innovation

• Restructures dataset into time-based series
• Uses Mann-Kendall test for performance trend
• Validates with 4 datasets and 5 models