AgentGuard: Runtime Verification of AI Agents

2025-09-28

AI Summary

Autonomous AI agents exhibit unpredictable emergent behaviors that invalidate conventional verification techniques. Method: This paper introduces a novel dynamic probabilistic assurance paradigm, which constructs formal event abstractions and Markov decision process (MDP) state models from runtime I/O observations, integrating online learning with probabilistic model checking to enable real-time, continuous, quantitative verification of agent behavior. Contribution/Results: The key innovation lies in embedding the dynamic probabilistic assurance mechanism directly into the runtime monitoring layer—enabling persistent evaluation of system failure probability under operational constraints. Experimental evaluation demonstrates that the framework delivers provably safe guarantees across diverse scenarios, while maintaining high responsiveness and strong robustness.

Technology Category

• Multiagent Systems: Agent-Based Simulation and Emergent Behavior
• Reasoning under Uncertainty: Sequential Decision Making
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Agentic search

Abstract

The rapid evolution to autonomous, agentic AI systems introduces significant risks due to their inherent unpredictability and emergent behaviors; this also renders traditional verification methods inadequate and necessitates a shift towards probabilistic guarantees where the question is no longer if a system will fail, but the probability of its failure within given constraints. This paper presents AgentGuard, a framework for runtime verification of Agentic AI systems that provides continuous, quantitative assurance through a new paradigm called Dynamic Probabilistic Assurance. AgentGuard operates as an inspection layer that observes an agent's raw I/O and abstracts it into formal events corresponding to transitions in a state model. It then uses online learning to dynamically build and update a Markov Decision Process (MDP) that formally models the agent's emergent behavior. Using probabilistic model checking, the framework then verifies quantitative properties in real-time.

Problem

• Verifying autonomous AI agents' runtime behavior probabilistically
• Providing continuous quantitative assurance for emergent behaviors
• Modeling agent behavior dynamically using online MDP learning

Innovation

• Dynamic Probabilistic Assurance for quantitative guarantees
• Online learning constructs Markov Decision Process model
• Probabilistic model checking verifies properties in real-time