Policy-Invisible Violations in LLM-Based Agents

2026-04-13

AI Summary

This work addresses “policy-invisible violations” in large language model agents—instances where outputs appear compliant but substantively breach organizational policies due to missing critical information such as entity attributes, contextual cues, or dialogue history. To tackle this, the authors propose Sentinel, a novel policy enforcement framework that introduces a world-state–based paradigm. Sentinel models agent actions as counterfactual update proposals over an organizational knowledge graph and validates structural invariants through speculative execution to precisely detect violations. The study also introduces PhantomPolicy, the first clean benchmark devoid of policy metadata, covering eight representative violation scenarios. Experiments demonstrate that Sentinel achieves 93.0% accuracy against human-audited ground-truth labels, substantially outperforming content-only DLP baselines (68.8%), thereby validating the efficacy of world-state modeling for accurate policy enforcement.

Technology Category

• Natural Language Processing: Safety and Robustness
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Agentic search
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

LLM-based agents can execute actions that are syntactically valid, user-sanctioned, and semantically appropriate, yet still violate organizational policy because the facts needed for correct policy judgment are hidden at decision time. We call this failure mode policy-invisible violations: cases in which compliance depends on entity attributes, contextual state, or session history absent from the agent's visible context. We present PhantomPolicy, a benchmark spanning eight violation categories with balanced violation and safe-control cases, in which all tool responses contain clean business data without policy metadata. We manually review all 600 model traces produced by five frontier models and evaluate them using human-reviewed trace labels. Manual review changes 32 labels (5.3%) relative to the original case-level annotations, confirming the need for trace-level human review. To demonstrate what world-state-grounded enforcement can achieve under favorable conditions, we introduce Sentinel, an enforcement framework based on counterfactual graph simulation. Sentinel treats every agent action as a proposed mutation to an organizational knowledge graph, performs speculative execution to materialize the post-action world state, and verifies graph-structural invariants to decide Allow/Block/Clarify. Against human-reviewed trace labels, Sentinel substantially outperforms a content-only DLP baseline (68.8% vs. 93.0% accuracy) while maintaining high precision, though it still leaves room for improvement on certain violation categories. These results demonstrate what becomes achievable once policy-relevant world state is made available to the enforcement layer.

Problem

• policy-invisible violations
• LLM-based agents
• organizational policy compliance
• contextual state
• entity attributes

Innovation

• policy-invisible violations
• PhantomPolicy
• Sentinel
• counterfactual graph simulation
• world-state-grounded enforcement