GuardianAgentBench: Where Agents Fail and How to Guard Them

2026-07-23

AI Summary

This work addresses the safety and reliability challenges faced by large language model (LLM) agents in autonomous tool use, where systematic evaluation and effective safeguards are currently lacking. The authors introduce GABench, a comprehensive benchmark comprising 580 scenarios across six domains, and evaluate six state-of-the-art models on three prominent frameworks—LangChain, LlamaIndex, and Vectara. Their analysis reveals two distinct failure modes in tool invocation, differentiating strong and weak models. To mitigate these issues, they propose a multi-stage verification approach combined with five types of adversarial attacks for robust evaluation, along with a runtime structured guardrail mechanism. Experiments show that even the best-performing configuration achieves only 74.8% overall accuracy; the proposed guardrails correct 19.9% of failure cases with a low false-positive rate of 0.5%, significantly outperforming baseline methods such as system prompting while preserving legitimate functionality.

Abstract

As large language model agents increasingly operate autonomously with access to tools and external environments, ensuring their safe and reliable behavior becomes critical. We present GuardianAgentBench (GABench), a benchmark of 580 scenarios across six domains evaluated on three production-ready frameworks: LangChain, LlamaIndex, and Vectara. The benchmark incorporates rigorous multi-stage validation and five adversarial attack modes. Experiments with six state-of-the-art models reveal that even the strongest configuration achieves only 74.8% overall accuracy and expose two distinct failure regimes: stronger models under-call required tools, while weaker models mis-select and over-call tools. Performance degrades monotonically with both tool-set size and sequential turn depth, with long-horizon planning proving the steeper bottleneck. Our guardrail implementation consistently outperforms system-prompt-based defenses across all models, recovering 19.9% of failures at a false positive rate of just 0.5%. These results demonstrate that execution-time structural intervention improves safety without disrupting correct agent behavior.

Problem

• agent safety
• tool usage failure
• autonomous agents
• adversarial robustness
• long-horizon planning

Innovation

• GuardianAgentBench
• adversarial attacks
• tool misuse
• guardrail mechanism
• execution-time intervention