Security-First Evaluation of Text-to-Terraform: Benchmarking LLMs and SLMs for Secure IaC Generation

2026-08-02

AI Summary

This study addresses the security risks arising from cloud misconfigurations by presenting the first systematic evaluation of large language models (LLMs) and small language models (SLMs) in generating secure and compliant Infrastructure-as-Code (IaC). Using 17 AWS Terraform scenarios, seven prominent models—including Claude Opus 4 and Qwen2.5-Coder-14B—were benchmarked under a pass@5 protocol within a GitLab CI/CD pipeline, employing Checkov and Trivy for security scanning across three compliance tiers and two prompting strategies. The findings reveal that syntactic correctness and security compliance are largely orthogonal: WizardCoder-33B achieved a 77.8% syntactic validity rate yet zero security compliance. Even with detailed security-oriented prompts, only Claude Opus 4 attained modest pass rates of 23.1% (Checkov) and 92.5% (Trivy), demonstrating that prompt engineering alone is insufficient for ensuring IaC security and underscoring the necessity of automated, multi-tool scanning.

Abstract

Cloud misconfiguration remains a leading cause of security incidents, yet whether LLMs and SLMs can generate security-compliant Infrastructure-as-Code is an open question. We benchmark seven models, three closed LLMs (Claude Opus 4, GPT-5.4, Gemini 2.5 Pro) and four open SLMs (Qwen2.5-Coder-14B, WizardCoder-33B, CodeLlama-13B, Magicoder-S-CL-7B), on AWS Terraform generation across 17 scenarios, integrating Checkov and Trivy scanners into a GitLab CI/CD pipeline and evaluating two prompt strategies at three security levels (pass@5). Syntactic validity and security compliance are largely orthogonal properties in LLM-generated IaC, a model that reliably produces well-formed Terraform does not necessarily produce secure Terraform: WizardCoder-33B achieves 77.8% validate rate yet zero Checkov compliance, while Claude Opus 4 reaches 23.1% Checkov and 92.5% Trivy pass rates under detailed security prompting. Consequently, prompt engineering alone is insufficient: automated multi-tool scanning remains a necessary complement to LLM-assisted IaC generation regardless of model family or prompt strategy. All artifacts are publicly available.

Problem

• Infrastructure-as-Code
• Security Compliance
• LLM
• Terraform
• Cloud Misconfiguration

Innovation

• Secure IaC Generation
• Security-First Evaluation
• Terraform
• LLM Benchmarking
• Automated Security Scanning