Where Did the Repair First Go Wrong? Localizing the Origins of Silent Failures in Agentic Vulnerability Repair

2026-10-05

AI Summary

This study addresses the silent failure problem in LLM agent-based vulnerability repair, where explicit error signals are absent. To overcome the limitations of conventional approaches that rely on such signals, this work proposes SAGE, a novel trace-based attribution framework. SAGE integrates security-aware gap assessment, multi-turn reasoning log analysis, and code history reconstruction to precisely identify the earliest reasoning turn at which the agent deviates from its intended security objective. Evaluated across 95 test cases, the method successfully localized the origins of failure in 93 instances. The findings reveal that the majority of silent failures stem not from inherent code generation deficiencies, but rather from unaddressed security requirements during the reasoning process.

Abstract

Localizing where an LLM-based agent first fails to uphold security during a repair can show which stage of its workflow needs an additional safeguard. This is difficult for silent failures, which are patches that pass syntactic and functional checks but still contain a security vulnerability. Because such patches give no observable failure signal, existing failure attribution methods, which rely on observed task failures and labelled failure steps, are less suited to them. We propose Security Awareness Gap Evaluation (SAGE), a trace-based method that combines an assessment of the security reasoning recorded at each turn with the reconstructed code history to identify the earliest turn at which a repair diverges from the task's security intent. We evaluate SAGE on 95 confirmed silent failures drawn from 3,684 repair traces produced by six agent frameworks and six base models on SecurityEval and CVEfixes. SAGE assigned an origin in 93 cases. Most origins were an unaddressed security requirement or an inadequate defence choice, and only five coincided with the code change itself. When the agent introduced the vulnerable code, the origin preceded the write in 14 of 19 cases. Repeated scoring and a second judge reproduced the origin type more consistently than the exact turn, and agreement was lowest for traces that kept only the final file.

Innovation

• Silent Failures
• Vulnerability Repair
• Failure Localization
• LLM Agents
• Security Reasoning