Towards a Risk Assessment of Malicious Skill Files in Coding Agents

2026-08-05

AI Summary

This work addresses a critical security vulnerability in coding agents whose capabilities are dynamically extended via skill files, yet whose natural language interfaces are susceptible to adversarial attacks disguised as benign instructions. We present the first systematic benchmark and reproducible evaluation framework for such interfaces, generating 2,826 high-fidelity malicious skills—mapped to MITRE ATT&CK tactics—using six large language models. Our evaluation pipeline integrates execution stratification, evidence anchoring, and intent coverage, complemented by multi-model adversarial generation, a three-judge LLM adjudication system, and human gold-standard validation. Across 5,629 trials, Gemini CLI and Qwen Code exhibited alarmingly high compromise rates of 95.5–96.1% and 71.6–74.0%, respectively, while explicit safety mechanisms achieved a mere 1.99% detection rate, exposing severe security flaws in current mainstream coding agents.

Abstract

Autonomous coding agents are increasingly embedded in enterprise software workflows with delegated authority over connected systems. Central to this architecture is the agent skills interface: folders of instructions and scripts that agents load dynamically to specialize their behavior. This interface also widens the attack surface, letting malicious shell commands hide within natural-language skill files. We make three contributions. First, an adversarial skill-synthesis method using six LLMs across four families to transform 471 real-world shell commands into benign-appearing skills, released as a benchmark of 2,826 skills mapped to 11 MITRE ATT&CK tactics. Second, a reproducible evaluation pipeline coupling run stratification, evidence anchoring, a refusal veto, and a deterministic declared-intent override with a three-judge LLM-as-a-judge panel, validated against a blind human gold standard (Cohen's kappa = 0.85). Third, a large-scale characterization of two enterprise-grade agents across 5,629 completed runs. Gemini CLI is exploited in 95.5-96.1% of runs and Qwen Code in 71.6-74.0% (raw majority vote to declared-intent-corrected estimate, both within the human gold standard), nearly invariant to the generating model. Explicit safety recognition occurs in only 1.99% of runs. Enterprises must assess and mitigate skill-interface risk before adopting coding agents. Our code and dataset are available at https://github.com/awsm-research/AgentJailbreak

Problem

• malicious skill files
• coding agents
• security risk
• attack surface
• MITRE ATT&CK

Innovation

• adversarial skill synthesis
• LLM-as-a-judge
• coding agent security
• skill interface risk
• MITRE ATT&CK benchmark