SkillSentry: Adaptive Honey Worlds for Dynamic Safety Testing of Agent Skills

2026-08-04

AI Summary

This work addresses the challenge that external skills invoked by large language model (LLM) agents may exhibit latent harmful behaviors under specific environmental conditions or interaction histories—risks that evade detection by existing static analysis methods. To tackle this, we propose SkillSentry, a dynamic security testing framework that simulates bait environments using LLMs, adaptively generates exploratory tasks, and compares execution trajectories with and without the target skill enabled. By correlating source code and runtime logs, SkillSentry enables precise attribution and detection of conditionally triggered malicious behaviors, overcoming the limitations of static approaches. Empirical evaluation shows that SkillSentry achieves 99.50% recall and an average F1 score of 96.26% on standard benchmarks. Notably, under semantic-preserving evasion attacks, it maintains robust performance with an average F1 of 92.95%, substantially outperforming the strongest baseline at 80.07%.

Abstract

External skills extend the capabilities of large language model agents, but also introduce an execution-time attack surface: a skill that appears benign under inspection may reveal harmful behavior only after particular environmental states, resources, or interaction histories are encountered. Existing scanners primarily rely on static analysis, predefined rules, or one-shot semantic judgments, making such conditional behavior difficult to elicit and attribute. We present SkillSentry, a dynamic safety-testing framework based on adaptive honey worlds. SkillSentry infers the intended capability boundary of a skill, constructs an LLM-simulated environment with controlled decoy resources, and adaptively generates tasks to explore its behavioral states. It then compares skill-enabled trajectories with matched no-skill executions, grounding suspicious behaviors in source code and verified execution traces before making a final decision. We evaluate SkillSentry against seven scanner configurations. SkillSentry achieves 99.50% Recall and 96.26% average F1 on standard benchmarks. Under semantics-preserving evasion, it reaches 92.95% average F1, compared with 80.07% for the strongest baselines. Our code is available at https://github.com/nizhangli062-jpg/SkillSentry-Adaptive-Honey-Worlds-for-Dynamic-Safety-Testing-of-Agent-Skills.

Problem

• agent safety
• conditional harmful behavior
• dynamic testing
• skill vulnerability
• execution-time attack surface

Innovation

• adaptive honey worlds
• dynamic safety testing
• agent skills
• behavioral state exploration
• execution trace analysis