AgentCanary: A Security Evaluation Framework for Autonomous AI Agents in Real Executable Environments

2026-06-09

AI Summary

Current safety evaluations of autonomous AI agents suffer from fragmented risk coverage, static or low-fidelity environments, and overly narrow metrics, rendering them inadequate against systemic safety threats. This work proposes a systematic evaluation framework centered on high-fidelity experimental environments: it first establishes an orthogonal Entry×Impact risk taxonomy aligned with realistic scenarios and a corresponding task suite; then constructs executable environments that support persistent states and multi-step interactions; and finally introduces a three-dimensional assessment mechanism—Outcome Safety, Security Awareness, and Task Utility—based on complete execution trajectories. Experiments across three major agent frameworks reveal pervasive deficiencies in security awareness under skill manipulation, state perturbation, and long-horizon attacks, establishing the first systematic benchmark for developing reliable AI agents.

Abstract

Autonomous AI agents have driven the transition from conversation to task execution, shifting security failures from textual deception to system compromise. Although security evaluation is crucial for proactive risk prevention, prior work is constrained by fundamental bottlenecks, including fragmented risk coverage, static or low-fidelity execution environments, and single-dimensional and coarse-grained assessment metrics. To address these challenges, we propose AgentCanary, a comprehensive security evaluation framework for autonomous AI agents. AgentCanary provides a systematic solution along three contributions. First, comprehensive risk coverage: we introduce an orthogonal Entry $\times$ Impact risk taxonomy that decouples how adversarial influence enters the agent from what harm it ultimately causes, and instantiate it as a scenario-aligned task suite spanning realistic deployment workflows. Second, a high-fidelity real executable environment: rather than static Q&A or mocked tool responses, agents interact with real tools against dynamically provisioned task artifacts, with persistent state across multi-step interactions that naturally supports long-horizon attack evaluation. Third, trajectory-grounded multi-dimensional evaluation: evaluation consumes the full agent trajectory rather than the reply text or a single tool call, enabling decomposed scoring along three orthogonal dimensions, Outcome Safety, Security Awareness, and Task Utility. We evaluate a broad set of frontier models on AgentCanary against multiple established adversarial attack methods across three agent frameworks. The results reveal that current agents often fail to recognize the attacks they face, particularly under compromised skills, persistent state, and long-horizon execution attacks, and provide a systematic baseline for developing more reliable and secure agent systems.

Problem

• autonomous AI agents
• security evaluation
• executable environments
• adversarial attacks
• risk coverage

Innovation

• security evaluation
• autonomous AI agents
• real executable environment
• risk taxonomy
• trajectory-based assessment