SkillHarm: Lifecycle-Aware Skill-Based Attacks via Automated Construction

2026-06-01

AI Summary

Third-party skills introduce a high-risk attack surface in agent workflows, yet existing research lacks systematic evaluation of their end-to-end lifecycle vulnerabilities and automated exploitation techniques. This work proposes SkillHarm, the first benchmark that establishes a comprehensive 12-category risk taxonomy spanning the skill lifecycle, and introduces two novel attack paradigms: fixed-payload poisoning and self-mutating poisoning. Leveraging the natural language–guided automation framework AutoSkillHarm—combined with skill package tampering and content-persistence mutation techniques—the authors generate 879 attack samples targeting 71 distinct skills. Experiments reveal alarming success rates of 86.3% and 69.3% against current agents under the two attack types, respectively. Moreover, most apparent “defenses” merely reflect failure to trigger malicious behavior rather than genuine mitigation, exposing the unreliability of existing protection mechanisms.

Abstract

Agent skills occupy a privileged position in the agent workflow, as agents are expected to implicitly follow and execute them, rendering third-party skills a vulnerable attack surface. Existing studies have revealed unsafe agent behaviors induced by skill-based attacks, but they primarily evaluate poisoned skills within a single task execution and enumerate harms through ad-hoc risk lists. To bridge these gaps, we introduce SkillHarm, a benchmark of skill-based attacks across the skill-use lifecycle, paired with a systematic taxonomy of skill-relevant risks. SkillHarm evaluates two attack scenarios: Fixed-Payload Poisoning (FPP), where a fixed poisoned skill package directly compromises any task session that invokes it, and Self-Mutating Poisoning (SMP), where an initially benign execution silently mutates persistent skill content, deferring harm until a subsequent reuse. It further defines 12 risk types based on the agent workflow component targeted by the harm: data pipelines, system environments, and agent autonomy. To instantiate these attacks at scale, we build AutoSkillHarm, an automated construction pipeline with coding agents driven by natural-language harnesses. The resulting benchmark contains 879 attack samples across 71 skills. Experiments show that current agents remain vulnerable with attack success rates up to 86.3% in FPP and 69.3% in SMP. Our analysis further reveals a latent risk: many apparent attack failures stem from the agent failing to engage with the poisoned file rather than genuine resistance, and current defenses still fail to reliably mitigate the threat.

Problem

• skill-based attacks
• agent workflow
• attack lifecycle
• risk taxonomy
• poisoned skills

Innovation

• skill-based attacks
• lifecycle-aware
• automated construction
• agent poisoning
• risk taxonomy