Agentic AI Security: Threats, Defenses, Evaluation, and Open Challenges

2025-10-27

AI Summary

This paper addresses novel security risks emerging from LLM-driven autonomous AI agents—systems capable of planning, tool invocation, and cross-environment execution. We propose the first systematic security risk taxonomy specifically designed for agent-based systems. Unlike conventional AI or software security frameworks, our taxonomy comprehensively covers multi-dimensional attack surfaces across networked, software, and physical environments. It integrates threat modeling, security evaluation benchmarks, defense mechanism analysis, and governance considerations, exposing critical risks overlooked by existing approaches. Crucially, this work is the first to structurally characterize agent security risks through both technical and governance lenses, systematically identifying practical attack/defense gaps and evaluation bottlenecks. We delineate key open challenges and provide foundational insights to guide secure-by-design agent architectures, standardization efforts, and future system evolution. (149 words)

Technology Category

• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Multiagent Systems: Agent/AI Theories and Architectures
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Agentic search
• Security and Privacy: Security and privacy of machine learning and AI applications

Abstract

Agentic AI systems powered by large language models (LLMs) and endowed with planning, tool use, memory, and autonomy, are emerging as powerful, flexible platforms for automation. Their ability to autonomously execute tasks across web, software, and physical environments creates new and amplified security risks, distinct from both traditional AI safety and conventional software security. This survey outlines a taxonomy of threats specific to agentic AI, reviews recent benchmarks and evaluation methodologies, and discusses defense strategies from both technical and governance perspectives. We synthesize current research and highlight open challenges, aiming to support the development of secure-by-design agent systems.

Problem

• Identifies security threats unique to autonomous AI agents
• Evaluates defense strategies and benchmarks for agentic systems
• Highlights open challenges in secure-by-design agent development

Innovation

• Taxonomy of agentic AI threats and defenses
• Benchmarks for evaluating agentic AI security
• Secure-by-design agent system development strategies