IssueTrojanBench: Benchmarking AI Coding Agents Against Malicious Issue Requests

2026-07-22

AI Summary

Current AI coding agents exhibit weak defenses against maliciously crafted issue requests, often being manipulated into generating harmful code or leaking sensitive data. This work proposes IssueTrojanBench, a novel benchmark that formally defines four new attack patterns and six delivery vectors, and incorporates a perturbation-enhanced mechanism to systematically evaluate the security performance of mainstream coding agents—including Cursor, Claude Code, and Codex Desktop—powered by GPT-5.3/5.4 and Sonnet 4.6 models. Experimental results reveal that 66.5% of malicious requests fully bypass existing safeguards, with the underlying large language models (LLMs) serving as the primary line of defense while agent-level protections prove largely ineffective. Notably, Sonnet 4.6 demonstrates superior risk awareness, underscoring the critical inadequacies in current safety mechanisms for AI coding assistants.

Abstract

AI coding agents powered by LLMs are increasingly integrated into real-world software development, where they generate, edit, and execute code with autonomous access to local files and tools. Coding agents inherit security risks from both the LLM backbone, where adversarial prompts, poisoned training data, and backdoor triggers can cause models to emit insecure or attacker-chosen code, and their agentic architecture, where tool-using autonomy enables induced misuse of external APIs, data exfiltration, and persistent compromise of development environments. This paper presents a systematic evaluation of malicious issue requests against state-of-the-art coding agents (Cursor, Claude Code, and Codex Desktop), powered by two major model families (OpenAI GPT-5.3 Codex/GPT-5.4 and Anthropic Sonnet 4.6). Our novel benchmark IssueTrojanBench contains malicious issues that are constructed based on four novel attack categories (i.e., embedded as malicious instructions in issues), six delivery vectors (e.g., PDF, or issue comment), and further augmented by perturbations. Our results reveal critical vulnerabilities in the as-deployed modern coding agents, i.e., 66.5% of the malicious issues from IssueTrojanBench penetrate all the guardrails (agent- and LLM-level) of coding agents. Our further analysis shows that rejection is almost entirely from LLMs rather than the agent frameworks, with GPT models broadly vulnerable and Sonnet 4.6 exhibiting more selective, risk-aware blocking of high-impact actions. Our evaluation also highlights that the current agent-level defense strategy offers limited additional protection for coding agents. Our findings highlight the urgent need for stronger agent- and model-level safety mechanisms to protect AI coding agents.

Problem

• AI coding agents
• malicious issue requests
• security vulnerabilities
• LLM safety
• agent-level defense

Innovation

• IssueTrojanBench
• AI coding agents
• malicious issue requests
• security benchmarking
• LLM vulnerabilities