ALIBI: Adaptive Agentic Attacks on LLM-Based Vulnerability Detectors via Adversarial Code Comments

2026-07-27

AI Summary

This work demonstrates that large language model (LLM)-based vulnerability detectors are highly susceptible to adversarial code comment attacks—crafted comments that leave program semantics unchanged yet effectively evade detection. To address this, the authors introduce ALIBI, the first framework to systematically treat code comments as an attack surface, proposing an agent-based adaptive black-box attack methodology. ALIBI employs multiple agents simulating developer behaviors and iteratively refines adversarial comments using feedback from the target detector. Evaluated on 125 real-world null pointer dereference vulnerabilities, the attack achieves over 90% success rate, reaching up to 100% in certain settings. The study further shows that defense strategies combining comment sanitization and architectural isolation significantly mitigate such attacks.

Abstract

Large language models are increasingly deployed for security-sensitive tasks such as vulnerability detection and code review. Their reliance on natural-language context embedded in source code exposes a previously underexplored attack surface: adversarial comments that can influence a detector's reasoning without changing program behavior. We study LLM-based vulnerability detectors against a new adversary: a coding agent that implements new functionality, deliberately introduces vulnerabilities, and strategically inserts adversarial source-code comments to evade detection. We present ALIBI, an automated adaptive black-box attack framework that generates and iteratively refines adversarial comments using detector reasoning and feedback. We transform real-world vulnerability-fixing commits into coding tasks and evaluate four representative LLM-based vulnerability detectors, ranging from specialized open-weight reasoning models to frontier multi-agent systems. All evaluated detectors are highly vulnerable: attack success rates exceed 90% across 125 real-world null-pointer dereference vulnerabilities, reaching 100% on one system. The framework also generalizes beyond this vulnerability class. Adversarial comments steering detector reasoning or fabricating external tool results prove most effective, while iterative refinement based on detector feedback further increases attack success. Finally, prompt-level defenses provide limited robustness against adaptive attacks, whereas architectural isolation and pre-detector comment sanitization substantially improve resilience. Our findings expose a fundamental attack surface in current LLM-based vulnerability detectors and motivate security-aware designs that carefully calibrate trust between natural-language context and program evidence.

Problem

• adversarial attacks
• LLM-based vulnerability detectors
• adversarial code comments
• security-sensitive tasks
• attack surface

Innovation

• adversarial code comments
• LLM-based vulnerability detection
• adaptive black-box attack
• agentic attack
• comment sanitization