Adaptive Hierarchical Evaluation of LLMs and SAST tools for CWE Prediction in Python

2026-01-04

arXiv.org

AI Summary

This work addresses the limitations of existing code vulnerability detection benchmarks, which predominantly rely on binary classification and lack fine-grained Common Weakness Enumeration (CWE)–level feedback necessary for iterative repair. To bridge this gap, we introduce ALPHA, the first function-level Python vulnerability benchmark, accompanied by a hierarchical CWE taxonomy and an adaptive penalty mechanism that distinguishes between over-generalization, over-specificity, and lateral misclassification errors. This framework enables, for the first time, fine-grained quantitative evaluation of CWE prediction quality. Using ALPHA, we systematically assess seven large language models (LLMs) and two static application security testing (SAST) tools, revealing that LLMs generally outperform SAST tools—despite the latter’s higher precision—and exhibit substantial variation in prediction consistency, ranging from 8.26% to 81.87%.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Computer Vision: Large Vision Models
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Security and Privacy: Large-scale security measurements

Abstract

Large Language Models have become integral to software development, yet they frequently generate vulnerable code. Existing code vulnerability detection benchmarks employ binary classification, lacking the CWE-level specificity required for actionable feedback in iterative correction systems. We present ALPHA (Adaptive Learning via Penalty in Hierarchical Assessment), the first function-level Python benchmark that evaluates both LLMs and SAST tools using hierarchically aware, CWE-specific penalties. ALPHA distinguishes between over-generalisation, over-specification, and lateral errors, reflecting practical differences in diagnostic utility. Evaluating seven LLMs and two SAST tools, we find LLMs substantially outperform SAST, though SAST demonstrates higher precision when detections occur. Critically, prediction consistency varies dramatically across models (8.26%-81.87% agreement), with significant implications for feedback-driven systems. We further outline a pathway for future work incorporating ALPHA penalties into supervised fine-tuning, which could provide principled hierarchy-aware vulnerability detection pending empirical validation.

Problem

• CWE prediction
• vulnerability detection
• LLMs
• SAST tools
• hierarchical evaluation

Innovation

• hierarchical evaluation
• CWE-specific penalties
• LLM vulnerability prediction
• adaptive benchmarking
• SAST comparison