OpenSkillRisk: Benchmarking Agent Safety When Using Real-World Risky Third-Party Skills

2026-07-22

AI Summary

This work addresses the latent runtime security risks faced by large language model agents when invoking third-party tools—a challenge inadequately mitigated by current systems. To this end, we introduce OpenSkillRisk, the first benchmark specifically designed to evaluate agent safety against real-world third-party skills, encompassing seven threat categories and 263 risky tools, supported by standardized tasks and a sandboxed execution environment. The benchmark enables fine-grained behavioral analysis across three major CLI-based agent frameworks and thirteen large language models. Our experiments reveal pervasive issues including contextual misjudgment and insufficient system-level risk awareness, with even optimally configured agents executing unsafe operations in approximately 17% of cases, exposing three distinct failure modes.

Abstract

LLM-based agents leverage third-party skills to extend their capabilities in open-world scenarios. However, third-party skills can introduce extra security vulnerabilities, as seemingly harmless skills can contain latent safety risks that only emerge during actual execution. In this work, we conduct a systematic investigation into how well current agent systems recognize and avoid such risks. To support quantitative and qualitative evaluation, we construct OpenSkillRisk, a dedicated safety benchmark containing 263 risky skills collected from public skill marketplaces. We classify these skills into seven categories based on their threat types and pair each skill with a standardized user task and a corresponding sandbox for controlled evaluation. Distinct from prior benchmarks, OpenSkillRisk not only covers more realistic and diverse unsafe scenarios, but also provides a fine-grained analysis to diagnose the behavioral patterns of agents in such scenarios. We conduct comprehensive experiments covering three mainstream CLI agent frameworks and thirteen state-of-the-art LLMs. Experimental results show that no tested system handles risky skills reliably: even the safest configurations still execute unsafe actions in about 17% of cases. Context-dependent and system-level risks are especially difficult for current agent systems to avoid. Our behavioral analysis reveals three recurring failure patterns: agents may fail to recognize the risk, recognize it but fail to intervene before acting, or follow skill instructions beyond the user's intended scope. These findings highlight the need to improve both risk reasoning in LLMs and execution control in agent frameworks.

Problem

• agent safety
• third-party skills
• security vulnerabilities
• risk recognition
• LLM-based agents

Innovation

• OpenSkillRisk
• agent safety
• third-party skills
• risk benchmarking
• LLM-based agents