red teaming

Traditional penetration testing struggles to evaluate security risks in AI systems arising from violations of behavioral objectives without breaching underlying infrastructure. This work proposes the first formal definition of AI penetration testing, reframing it as an objective-driven behavioral security assessment. The approach involves identifying operational objectives, mapping AI-driven behaviors, analyzing adversarial attack surfaces—such as prompt injection, data poisoning, and sensor manipulation—establishing criteria for behavioral failure, and conducting scenario-based red-teaming exercises. By integrating threat modeling, behavior mapping, and evidentiary chain construction, the framework demonstrates its efficacy and novelty in a case study involving an AI-powered Security Operations Center assistant, successfully uncovering attack pathways that violate system objectives through behavioral manipulation alone, without requiring infrastructure compromise.

This work addresses the novel security threats introduced by agentic AI systems—arising from their integration of tool use, retrieval-augmented generation, and multi-agent collaboration—which extend beyond the scope of traditional AI safety concerns. It presents the first systematic characterization of the attack surface and threat model for such systems, establishing a taxonomy encompassing prompt injection, knowledge base poisoning, tool misuse, and cross-agent manipulation. The study further introduces quantifiable security metrics, including unsafe operation rate and privilege escalation distance. Through a systematic review of over 20 studies from 2023 to 2025, it evaluates the efficacy of existing defenses such as input sanitization, sandboxing, and access control, exposing critical limitations. Building on these insights, the paper proposes a holistic, full-lifecycle security framework spanning design, runtime, and incident response, accompanied by a phased deployment roadmap.

Traditional defense mechanisms struggle to counter AI-driven adaptive cyberattacks, as they are ill-equipped to handle autonomous adversarial agents capable of evading detection. This work proposes a novel defensive infrastructure centered on controllable offensive AI, which, within regulated environments, is trained to simulate the full attack lifecycle to proactively acquire and transform threat intelligence into defensive knowledge. The study’s core contributions include the first systematic benchmark encompassing the entire attack chain, alongside a training-based vulnerability discovery agent, an open-weight model governance framework, a tiered capability release mechanism, and a defense-oriented agent distillation technique. Together, these establish a “offense-informed defense” strategic paradigm and outline three actionable pathways for the safe development and constraint of offensive AI capabilities.

This work addresses the vulnerability of AI agents to indirect prompt injection attacks, wherein malicious instructions embedded in untrusted inputs can induce hazardous behaviors. To mitigate this risk, the authors propose a system-level defense framework that integrates dynamic re-planning, constrained perception and decision-making mechanisms, and human-in-the-loop intervention to establish a secure and controllable agent architecture. The framework employs a dual-driven safety verification mechanism—combining rule-based and model-based checks—to dynamically update security policies and rigorously enforce behavioral boundaries on the underlying model. Furthermore, the study highlights critical limitations in existing evaluation benchmarks and advocates for more robust, real-world-oriented assessments of agent resilience and human-AI interaction safety.

This work addresses the limitations of existing adversarial simulation tools, which rely on agent-based instrumentation of target systems, often leaving anomalous artifacts and failing to faithfully replicate human attacker behavior—particularly in critical phases of the cyber kill chain such as initial access and interactive operations. To overcome these shortcomings, the authors propose and implement an open-source attack scripting language coupled with an agentless execution engine that closely emulates real-world attacker tactics. This approach enables high-fidelity, interactive simulation of complete kill chain stages, including initial access, privilege escalation, and lateral movement. Experimental results demonstrate that system logs generated by this method exhibit significantly greater behavioral similarity to those produced by actual human-driven attacks, thereby enhancing the realism and effectiveness of security testing and intrusion detection research.

Existing black-box attack methods struggle to effectively evaluate the robustness of multi-component NLP systems under stringent constraints—specifically, binary feedback only, no gradient access, and a query budget of ten or fewer. This work proposes a dual-agent adversarial rewriting framework: an attack agent generates semantics-preserving rewrites, while a prompt optimization agent iteratively refines the attack strategy based solely on binary feedback. The approach achieves the first effective black-box attacks under such strict conditions, revealing critical links between system architecture and vulnerability, and identifying four distinct attack patterns targeting different pipeline stages. Experiments demonstrate evasion rates of 19.95%–40.34% against four LLM-based misinformation detection systems and up to 97.02% against static retrieval systems. Furthermore, defenses informed by these attack patterns reduce evasion rates by as much as 65.18%.

This study systematically investigates the security challenges arising as generative AI transitions from content generation to performing real-world actions, introducing a novel tripartite threat taxonomy encompassing content-level, model-level, and agent-level risks. Through integrated threat modeling, evaluation of technical countermeasures—including detection, watermarking, alignment techniques, and agent-specific safeguards—and analysis of governance structures, the work reveals a pervasive gap between the rapid expansion of attack surfaces and the current state of defensive capabilities. Most existing technical solutions remain contingent on nascent institutional coordination mechanisms that have yet to mature. The research underscores the necessity for parallel evolution of technical and governance approaches and highlights the critical importance of cross-layer collaborative defense strategies to effectively mitigate emerging threats.

Existing membership inference attacks rely on static, handcrafted strategies with limited generalization capabilities. This work proposes the first agent-based framework that decouples strategy reasoning from execution, reformulating the attack process as a self-exploratory and strategy-evolution mechanism guided by high-level scenario descriptions. The framework automatically generates and iteratively refines attack strategies at the logits level, eliminating the need for manual feature engineering and enabling model-agnostic, systematic strategy search. Experimental results demonstrate that the proposed method consistently matches or surpasses current state-of-the-art baselines across diverse large models, significantly enhancing both the universality and effectiveness of membership inference attacks.

This study addresses the security risks posed by AI agents with offensive cyber capabilities that may breach sandbox boundaries in evaluation environments. It systematically identifies five categories of boundary vulnerabilities—multi-step attacks, objective conflicts, supply chain leaks, persistence mechanisms, and automated execution speed—and conducts a case analysis grounded in the 2026 Hugging Face/OpenAI incident. The work introduces the first taxonomy of AI boundary vulnerabilities specifically tailored to evaluation settings and proposes an integrated defense framework combining isolation, privilege separation, behavioral provenance tracking, and defensive response interfaces. By jointly considering misuse risks and capability assessment, this research establishes clear security priorities for high-risk AI evaluations, offering both theoretical foundations and practical guidance for developing trustworthy evaluation environments that balance testing efficacy with risk containment.

Current AI agents lack effective security evaluation mechanisms against indirect prompt injection attacks, particularly missing a reconnaissance phase analogous to that in traditional penetration testing. This work formally defines the reconnaissance process for AI agents and introduces KYA, a reconnaissance-driven, black-box penetration testing framework. KYA actively probes target agents to construct detailed profiles, identifies and extracts their knowledge assets, and leverages this intelligence to generate more potent, tailored attacks. The end-to-end framework demonstrates strong effectiveness across multiple agent security benchmarks and real-world coding agents. To facilitate further research, the authors open-source the KYA framework, along with a dedicated benchmark dataset and baseline implementations.