VulnGym: Benchmarking Coding Agents for Repository-Level Vulnerability Detection

2026-08-03

AI Summary

This work addresses the limitations of existing vulnerability detection benchmarks, which predominantly focus on preselected code snippets and thus fail to evaluate coding agents’ ability to autonomously locate vulnerabilities within full-scale code repositories. Moreover, these benchmarks lack fine-grained trajectory annotations necessary for diagnosing agent shortcomings. To bridge this gap, the authors propose VulnGym—an end-to-end benchmark grounded in real-world code repositories—by aligning GitHub Security Advisories with their corresponding vulnerable repository versions. VulnGym provides, for the first time, fine-grained annotations including line-level entry points, critical operations, and vulnerability propagation paths. The benchmark introduces three subtasks based on prior knowledge and a comprehensive evaluation framework. Experiments across 184 security advisories and 408 vulnerability instances reveal that current agents still exhibit significant deficiencies in repository-scale vulnerability localization and evidence chain construction.

Abstract

Recent advances in LLM-based vulnerability detection have shown promising results, while coding agents further extend this capability from isolated code snippets to complete repositories. This shift requires agents to autonomously explore repositories and locate vulnerability-relevant code, instead of performing detection on preselected functions. However, existing benchmarks primarily focus on vulnerability classification over preselected code snippets, limiting their ability to evaluate coding agents in repository-level vulnerability detection. Moreover, without fine-grained vulnerability trace annotations, the capability limitations underlying the detection process remain difficult to explore. To address these limitations, we present \textbf{VulnGym}, a real-world repository-level benchmark for evaluating vulnerability detection by coding agents. VulnGym aligns reviewed GitHub advisories with their corresponding vulnerable version repositories. It contains 184 advisories and 408 vulnerability entries across 23 repositories, with each entry annotated with line-level entry points, critical operations, and vulnerability traces. Using this fine-grained ground truth, VulnGym defines an end-to-end detection task and three oracle-based subtasks to jointly evaluate vulnerability detection and diagnose limitations in code localization and evidence construction. Our evaluation indicates that current coding agents remain limited in both end-to-end repository-level vulnerability detection and the construction of accurate supporting traces.

Problem

• vulnerability detection
• coding agents
• repository-level
• benchmark
• vulnerability trace

Innovation

• repository-level vulnerability detection
• coding agents
• fine-grained annotation
• vulnerability trace
• LLM-based evaluation