Knowledge Graph Representations for LLM-Based Policy Compliance Reasoning

2026-04-30

AI Summary

This work addresses the compliance challenges posed by the rapid integration of artificial intelligence by proposing a knowledge graph–enhanced large language model (LLM) reasoning framework that efficiently and accurately extracts and reasons over compliance information from complex policy documents. The approach employs an agent-based architecture to automatically construct policy knowledge graphs and compares manually defined ontologies with those automatically discovered by LLMs, revealing the latter’s superior performance in cross-policy reasoning tasks. Comprehensive experiments across six question-answering categories, five LLMs, and multiple evaluation strategies consistently demonstrate—across 42 tasks—that integrating knowledge graphs significantly enhances LLMs’ compliance reasoning capabilities while exhibiting strong generalization.

Abstract

The risks posed by AI features are increasing as they are rapidly integrated into software applications. In response, regulations and standards for safe and secure AI have been proposed. In this paper, we present an agentic framework that constructs knowledge graphs (KGs) from AI policy documents and retrieves policy-relevant information to answer questions. We build KGs from three AI risk-related polices under two ontology schemas, and then evaluate five LLMs on 42 policy QA tasks spanning six reasoning types, from entity lookup to cross-policy inference, using both heuristic scoring and an LLM-as-judge. KG augmentation improves scores for all five models, and an open, LLM-discovered schema matches or exceeds the formal ontology.

Problem

• Knowledge Graph
• LLM
• Policy Compliance
• AI Regulation
• Reasoning

Innovation

• knowledge graph
• LLM-based reasoning
• policy compliance
• ontology schema
• AI regulation