🤖 AI Summary
AI governance policies—typically authored in natural language—are manually translated into executable rules, resulting in low efficiency, high error rates, and poor scalability, thereby impeding the deployment of safety mechanisms. Method: We propose Policy-to-Tests (P2T), the first systematic framework to automatically convert multi-source AI policy documents into standardized, machine-executable rules. P2T introduces a compact domain-specific language (DSL) to structurally encode policy elements—including risk categories, scope, conditions, exceptions, and evidentiary requirements—and employs an LLM-driven parsing and adjudication pipeline for end-to-end policy interpretation and behavioral compliance assessment. Contribution/Results: Experiments show P2T-generated rules match human-authored baselines in coverage and granularity (high inter-annotator agreement); when integrated with HIPAA-aligned safeguards, P2T significantly reduces agent policy violations. All artifacts—including source code, DSL specification, prompt templates, and rule sets—are publicly released to ensure reproducibility.
📝 Abstract
AI policy guidance is predominantly written as prose, which practitioners must first convert into executable rules before frameworks can evaluate or enforce them. This manual step is slow, error-prone, difficult to scale, and often delays the use of safeguards in real-world deployments. To address this gap, we present Policy-to-Tests (P2T), a framework that converts natural-language policy documents into normalized, machine-readable rules. The framework comprises a pipeline and a compact domain-specific language (DSL) that encodes hazards, scope, conditions, exceptions, and required evidence, yielding a canonical representation of extracted rules. To test the framework beyond a single policy, we apply it across general frameworks, sector guidance, and enterprise standards, extracting obligation-bearing clauses and converting them into executable rules. These AI-generated rules closely match strong human baselines on span-level and rule-level metrics, with robust inter-annotator agreement on the gold set. To evaluate downstream behavioral and safety impact, we add HIPAA-derived safeguards to a generative agent and compare it with an otherwise identical agent without guardrails. An LLM-based judge, aligned with gold-standard criteria, measures violation rates and robustness to obfuscated and compositional prompts. Detailed results are provided in the appendix. We release the codebase, DSL, prompts, and rule sets as open-source resources to enable reproducible evaluation.