Vexed by VEX tools: Consistency evaluation of container vulnerability scanners

2025-03-18

AI Summary

This paper addresses the critical lack of consistency in Vulnerability Exploitability eXchange (VEX) format adoption across container vulnerability scanners. Method: We conduct the first systematic evaluation of mainstream VEX tools—not for isolated accuracy, but for their collaborative capacity to refine Software Bill of Materials (SBOM)-based security alerts. We propose a quantitative, cross-tool and cross-image comparison framework grounded in Jaccard and Tversky similarity metrics, validated empirically on a standardized container image benchmark dataset. Results: VEX reports exhibit low inter-tool consistency, exposing immaturity in the current VEX ecosystem. Root-cause analysis fails to isolate fundamental bottlenecks, underscoring urgent needs for methodological alignment and specification harmonization. This work establishes the first reproducible evaluation paradigm and empirical benchmark to guide VEX standardization and enhance tool interoperability.

Technology Category

• Constraint Satisfaction and Optimization: Solvers and Tools
• Computer Vision: Adversarial Attacks & Robustness
• Search and Optimization: Evaluation and Analysis

Application Category

• Security and Privacy: Large-scale security measurements
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization

Abstract

This paper presents a study that analyzed state-of-the-art vulnerability scanning tools applied to containers. We have focused the work on tools following the Vulnerability Exploitability eXchange (VEX) format, which has been introduced to complement Software Bills of Material (SBOM) with security advisories of known vulnerabilities. Being able to get an accurate understanding of vulnerabilities found in the dependencies of third-party software is critical for secure software development and risk analysis. Accepting the overwhelming challenge of estimating the precise accuracy and precision of a vulnerability scanner, we have in this study instead set out to explore how consistently different tools perform. By doing this, we aim to assess the maturity of the VEX tool field as a whole (rather than any particular tool). We have used the Jaccard and Tversky indices to produce similarity scores of tool performance for several different datasets created from container images. Overall, our results show a low level of consistency among the tools, thus indicating a low level of maturity in VEX tool space. We have performed a number of experiments to find and explanation to our results, but largely they are inconclusive and further research is needed to understand the underlying causalities of our findings.

Problem

• Evaluate consistency of VEX-based container vulnerability scanners
• Assess maturity of VEX tools in vulnerability detection
• Analyze tool performance using Jaccard and Tversky indices

Innovation

• Analyzed VEX format vulnerability scanning tools
• Used Jaccard and Tversky indices for consistency
• Assessed low maturity in VEX tool space