An Automated Framework for Cybersecurity Policy Compliance Assessment Against Security Control Standards

2026-05-08

AI Summary

This study addresses the challenges of assessing compliance between organizational cybersecurity policies and abstract security control frameworks such as NIST SP 800-53, which are often time-consuming, difficult to standardize, and lack traceability. To overcome these limitations, the authors propose PROPAGATE, a novel framework that leverages large language models (LLMs) to automate control-level compliance evaluation for the first time. By integrating both open-source and closed-source LLMs, the framework automatically retrieves relevant policy text, evaluates coverage across 1,007 security controls, and generates interpretable gap analyses with actionable improvement recommendations. Experimental results on two real-world organizational policy corpora demonstrate high effectiveness, achieving F1 scores of 88.54 and 82.31, respectively, thereby enabling traceable and explainable compliance enhancement.

Abstract

Organizational cybersecurity policies are often examined to determine whether they adequately comply standard security controls. This task is difficult because control statements are abstract, whereas policy documents describe governance practices in varied natural language. As a result, policy-based control assessment is time-consuming, difficult to standardize, and often difficult to document in a traceable manner. To address this gap, we present PROPARAG, an audit support approach for evaluating organizational cybersecurity policies against security controls autonomously. For each control, the approach retrieves relevant policy evidence, assesses coverage, identifies missing elements, and generates supporting explanations and recommendations. We evaluate PROPARAG on two real-world organizational policy corpora using 1,007 NIST SP 800-53 controls across both closed-source and open-source large language models (LLMs). The framework achieves F1 scores of 88.54 on OrgA and 82.31 on OrgB. The evaluation also shows that PROPARAG identifies relevant gaps in documented organizational policies and generates grounded recommendations for each identified gap. This research provides foundation for LLM-powered autonomous control-level assessment of organizational cybersecurity policies.

Problem

• cybersecurity policy
• compliance assessment
• security controls
• policy evaluation
• NIST SP 800-53

Innovation

• automated compliance assessment
• large language models
• cybersecurity policy
• security controls
• NIST SP 800-53