SecProbe: Adaptive Evaluation of Coding Agents on Cybersecurity Vulnerabilities

2026-09-27

AI Summary

This study addresses the limitations of static benchmarks in adapting to model evolution and the scarcity of vulnerability data by proposing a dynamic evaluation framework that integrates Item Response Theory (IRT) with on-demand synthesis techniques. Through automated vulnerability task synthesis and a multilingual code agent architecture, the framework enables adaptive assessment of coding agents' secure repair capabilities. Its core innovation lies in an IRT-based dynamic task selection mechanism that significantly reduces computational overhead while preserving evaluation precision. Experimental evaluation across 353 constructed vulnerability repair tasks reveals that state-of-the-art models achieve a success rate of only 28.33%, while the proposed approach reduces computational costs by 29.5%.

Abstract

Assessing cybersecurity vulnerability awareness in coding agents requires evaluations that reveal capability gaps and remain informative as models evolve. Static benchmarks offer fixed coverage and difficulty, while scarce vulnerable repositories and costly expert authoring limit their renewal at scale. We introduce SecProbe, a framework for adaptive evaluation that combines Item Response Theory (IRT) with on-demand synthesis of repository-scale vulnerability-repair tasks. From observed performance, \textsc{SecProbe} estimates agent ability and identifies where additional evidence is most informative, selecting existing tasks or synthesizing new ones accordingly. As one use case, we construct 353 tasks spanning six programming languages and 151 CWE types and evaluate nine frontier models with two agent harnesses. Success rates peak at 28.33\%, highlighting substantial gaps in vulnerability recognition and repair. Compared with random and one-shot baselines, \textsc{SecProbe} achieves comparable agent ability estimates while requiring agents to solve up to 29.5\% fewer tasks. These results support adaptive evaluation as an efficient and discriminative approach to assessing cybersecurity vulnerability awareness.

Problem

• coding agents
• cybersecurity vulnerabilities
• adaptive evaluation
• vulnerability repair
• benchmarking

Innovation

• Adaptive Evaluation
• Item Response Theory
• Coding Agents
• Cybersecurity Vulnerabilities
• On-demand Synthesis