Runtime Governance for AI Agents: Policies on Paths

2026-03-17

AI Summary

This work addresses the challenge of governing AI agents whose runtime behaviors exhibit nondeterminism and path dependence, rendering traditional design-phase governance mechanisms insufficient. It proposes a novel runtime governance framework centered on execution paths, formalizing compliance policies as deterministic functions of agent identity, partial execution history, next action, and organizational state—thereby enabling dynamic oversight of path-dependent behaviors for the first time. The framework subsumes system prompts and static access control as special cases and establishes runtime evaluation as a general governance paradigm. Integrating large language model–driven behavior analysis and policy assessment, it offers AI Act–inspired policy examples and reference implementations while highlighting critical open issues such as risk calibration and the boundaries of enforceable compliance.

Technology Category

• Philosophy and Ethics of AI: AI & Law, Justice, Regulation & Governance
• Multiagent Systems: Mechanism Design
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Agentic search

Abstract

AI agents -- systems that plan, reason, and act using large language models -- produce non-deterministic, path-dependent behavior that cannot be fully governed at design time, where with governed we mean striking the right balance between as high as possible successful task completion rate and the legal, data-breach, reputational and other costs associated with running agents. We argue that the execution path is the central object for effective runtime governance and formalize compliance policies as deterministic functions mapping agent identity, partial path, proposed next action, and organizational state to a policy violation probability. We show that prompt-level instructions (and "system prompts"), and static access control are special cases of this framework: the former shape the distribution over paths without actually evaluating them; the latter evaluates deterministic policies that ignore the path (i.e., these can only account for a specific subset of all possible paths). In our view, runtime evaluation is the general case, and it is necessary for any path-dependent policy. We develop the formal framework for analyzing AI agent governance, present concrete policy examples (inspired by the AI act), discuss a reference implementation, and identify open problems including risk calibration and the limits of enforced compliance.

Problem

• AI agents
• runtime governance
• path-dependent behavior
• compliance policies
• non-deterministic execution

Innovation

• runtime governance
• execution path
• compliance policy
• AI agents
• path-dependent behavior