SeClaw: Spec-Driven Security Task Synthesis for Evaluating Autonomous Agents

2026-06-01

AI Summary

Existing safety evaluation methods for autonomous agents predominantly rely on manually designed tasks, which offer limited coverage and focus solely on final outputs, thereby failing to capture unsafe behaviors that emerge during execution in complex environments. To address this limitation, this work proposes SeClaw, a novel framework that, for the first time, automatically generates safety evaluation tasks from structured risk specifications. By integrating Docker-based containerized testing environments with a trajectory-aware assessment mechanism, SeClaw enables fine-grained, reproducible safety evaluations of agent behavior across multiple risk dimensions—including resources, tasks, environment, and self-conducted actions—throughout the entire execution trajectory. The framework establishes a standardized benchmark encompassing a broad spectrum of safety threats, providing a measurable, diagnosable, and comparable foundation for evaluating LLM-based agents.

Abstract

Autonomous LLM agents increasingly operate in stateful environments where they access tools, files, memory, and external services. While such capabilities enable complex real-world workflows, they also introduce security risks that are difficult to capture with existing evaluations. Current agent security benchmarks often rely on manually curated tasks, provide limited coverage of emerging threats, and focus primarily on final outcomes rather than the execution processes that lead to unsafe behavior. We introduce SeClaw, a framework that combines specification-driven security task synthesis with execution-based security evaluation for Autonomous agents. Spec-driven security task synthesis enables scalable and controllable construction of security tasks from structured risk specifications, while SeClaw docker provides a standardized testbed for evaluating agent behavior under diverse safety-risk scenarios. The benchmark covers risks arising from resources, user tasks, environments, and intrinsic agent behaviors, and supports trajectory-aware assessment of unsafe actions beyond final responses. By bridging systematic task synthesis and reproducible security evaluation, SeClaw provides a practical foundation for measuring, diagnosing, and comparing security failures in autonomous LLM agents. The code is available at https://github.com/seclaw-eval/seclaw-eval.

Problem

• autonomous agents
• security evaluation
• LLM safety
• task synthesis
• execution trajectory

Innovation

• specification-driven synthesis
• autonomous agents
• security evaluation
• trajectory-aware assessment
• LLM safety