MalSkillBench: A Runtime-Verified Benchmark of Malicious Agent Skills

2026-06-05

AI Summary

This work addresses the lack of runtime verification benchmarks in the hybrid space of code and natural language, which hinders effective detection of malicious AI agent skills. The authors propose MalSkillBench, the first runtime benchmark for malicious skills, featuring a three-dimensional evaluation framework that integrates code, instructions, and task intent. It establishes a Generate-Verify-Feedback loop through Docker sandbox execution, system call monitoring, and large model–based adjudication, and includes 3,944 annotated malicious skill samples combining real-world and synthetically generated data. Experiments show a 98.4% detection rate for code injection attacks but only 75.8% for prompt injection attacks. Wild samples predominantly target cryptocurrency theft, and joint analysis of code and instructions proves essential for effective detection, exposing significant blind spots in current methods regarding prompt injection and agent control-plane attacks.

Abstract

AI coding agents such as Claude Code and Gemini CLI increasingly extend themselves with third-party skills: markdown packages bundling natural-language instructions, executable scripts, and tool permissions. Because a skill is at once code and agent-facing instruction, it introduces a supply chain dependency whose risk is neither pure code nor pure prompt. Detection tools have never been measured against verified ground truth spanning this hybrid space, leaving their effectiveness unknown and wild-only evaluations biased. We present MalSkillBench, the first runtime-verified benchmark of malicious agent skills: 3,944 malicious skills labeled along a three-dimensional taxonomy of 108 cells. Of these, 3,214 come from a closed-loop Generate-Verify-Feedback pipeline admitting only samples whose malicious behavior fires inside a Docker sandbox under system-call monitoring and an LLM judge; we add 703 in-the-wild and 4,000 matched benign skills. Our measurements are consistent: code injection reaches 94.5% verification yield but prompt injection only 75.8%, the same fragility that later makes it hard to detect; the wild sample is narrow, dominated by one cryptocurrency-theft campaign (86.6% one behavior, 81% from two accounts) with a small but architecturally new tail attacking the agent control plane; the strongest skill-specific detector reaches 98.4% recall on code injection yet collapses on prompt-injection and agent-control attacks, and wild-only scoring swings the ranking by up to 66 recall points; supply-chain scanners and prompt-injection defenses each see only half of a skill, and no combination recovers the code-instruction relationship. Detecting malicious skills therefore requires reasoning jointly over task intent, code, and instructions. We release the dataset, pipeline, baselines, and results.

Problem

• malicious agent skills
• runtime verification
• hybrid code-prompt threats
• benchmarking
• supply chain security

Innovation

• malicious agent skills
• runtime verification
• hybrid code-instruction threat
• skill supply chain
• benchmark