How Good LLM-Generated Password Policies Are?

2025-06-10

AI Summary

This study systematically evaluates the consistency, accuracy, and security of large language models (LLMs) in automatically translating natural-language password policy prompts into executable `pwquality.conf` configurations. We empirically assess mainstream open-source models—including Llama and Mixtral—under zero-shot prompting and official documentation–augmented prompting paradigms. Our methodology incorporates syntactic/semantic validation and cross-sample consistency metrics. Results reveal a >78% error rate under zero-shot prompting; even with documentation augmentation, 32% of outputs exhibit logical contradictions—highlighting fundamental LLM deficiencies in semantic understanding of security policies, constraint adherence, and generation stability for safety-critical configuration tasks. To address this gap, we propose the first verifiable policy-consistency evaluation framework and introduce the first empirical benchmark dedicated to password policy configuration generation. This work provides both methodological foundations and critical reliability warnings for AI-driven security automation.

Technology Category

Natural Language Processing: Safety and RobustnessMachine Learning: Large Multimodal Models (LMMs)Planning, Routing, and Scheduling: Planning with Language Models

Application Category

User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendationSecurity and Privacy: Large-scale security measurementsSearch and Retrieval-Augmented AI: Large language models for search

Abstract

Generative AI technologies, particularly Large Language Models (LLMs), are rapidly being adopted across industry, academia, and government sectors, owing to their remarkable capabilities in natural language processing. However, despite their strengths, the inconsistency and unpredictability of LLM outputs present substantial challenges, especially in security-critical domains such as access control. One critical issue that emerges prominently is the consistency of LLM-generated responses, which is paramount for ensuring secure and reliable operations. In this paper, we study the application of LLMs within the context of Cybersecurity Access Control Systems. Specifically, we investigate the consistency and accuracy of LLM-generated password policies, translating natural language prompts into executable pwquality.conf configuration files. Our experimental methodology adopts two distinct approaches: firstly, we utilize pre-trained LLMs to generate configuration files purely from natural language prompts without additional guidance. Secondly, we provide these models with official pwquality.conf documentation to serve as an informative baseline. We systematically assess the soundness, accuracy, and consistency of these AI-generated configurations. Our findings underscore significant challenges in the current generation of LLMs and contribute valuable insights into refining the deployment of LLMs in Access Control Systems.

Problem

• Assess consistency of LLM-generated password policies
• Evaluate accuracy of AI-created pwquality.conf files
• Improve LLM deployment in Access Control Systems

Innovation

• LLM-generated password policies evaluation
• Natural language to pwquality.conf translation
• Consistency and accuracy assessment methodology