🤖 AI Summary
This study reframes multi-agent AI safety as a problem of institutional design, focusing on security mechanisms in task delegation, information flow, action execution, and resource sharing. Through structured workflow experiments, it systematically links institutional components—such as rules, authority states, and fallback pathways—to emergent safety behaviors in multi-agent systems, revealing distinct failure modes under identical violation rates. The methodology integrates constitutional prompting, provenance-aware protection, and local state guards, evaluated across 5,280 high-conflict episodes involving multiple model families. Results show that both constitutional prompting and provenance-aware protection achieve zero violations out of 384 trials; the latter successfully blocks 51 attempted violations and ensures 44 subsequent safe completions. In contrast, local state guards exhibit significant failure under policy-change scenarios.
📝 Abstract
AI agents increasingly work inside systems that govern how they delegate tasks, move information, execute actions, and use shared resources. Recent work already shows that deployment rules can change collective behavior. Here we ask which parts of an AI institution produce safety and how they do it. This is the first paper from POLIS, an ongoing research programme studying algorithmic institutions for multi-agent systems. We report a frozen 5,280-episode study suite. The main pre-specified delegation experiment spans four model families; a targeted high-conflict diagnostic adds three additional model endpoints. In matched structured workflows, the model sees different rule formulations and guards consult different authority states. We also vary the attractiveness of the immediate compliant internal/self fallback and allow blocked workflows to continue. A detailed constitutional prompt produces 0/384 realized violations. A provenance-aware executable guard also produces 0/384, although it blocks prohibited attempts in 51/384 episodes; 44/51 of those episodes later complete safely. The local-state guard's failures concentrate in scenarios where an ordinary transformation changes visible policy while originating authority stays fixed. In matched laundering scenarios, that guard admits violations in 22/96 episodes and provenance enforcement in 0/96 (p = 4.77 x 10^-7). A separate resource-allocation experiment shows that revealing the numerical value of an otherwise identical cap changes agent requests. In these structured workflows, the same final violation rate can hide very different mechanisms. The rule itself is only part of the institution. The authority state the system trusts matters, and so does the path available after a block.