SecDrift: Measuring Sector-Conditioned Security Drift in AI-Generated Code

2026-07-27

AI Summary

This study investigates whether domain-specific prompts genuinely influence the security of code generated by large language models in critical infrastructure contexts. To this end, we introduce the SecDrift benchmark, which integrates static analysis, five-dimensional prompt perturbations, and matched baselines to systematically evaluate the impact of prompt context on security across eight domains and nine Common Weakness Enumeration (CWE) vulnerability categories. Employing mixed-effects logistic regression and Fisher’s exact tests for statistical analysis, our findings reveal no significant security drift attributable to industry-specific prompts (p = 0.24). Instead, observed differences in vulnerability rates—ranging from 11.6% to 16.1%—are primarily driven by model choice rather than prompt design. This work quantitatively demonstrates for the first time that purported “security improvements” stem from biases in vulnerability category composition, underscoring model selection as more critical than prompt engineering for secure code generation.

Abstract

LLMs are increasingly used for code generation in critical infrastructure, yet the security effect of domain-specific prompting is understudied. We present SecDrift, a benchmark measuring sector-conditioned security drift: the change in static-analysis vulnerability rates when prompts are conditioned on industry contexts versus neutral baselines. We evaluate 7 LLMs (6 producing analyzable code) across 8 CISA critical infrastructure sectors and 9 CWE categories with 5 replicates (5,355 evaluations), using a 5-dimension transformation with a matched-baseline condition that holds the task fixed while substituting only domain terminology. Industry prompts naively appear more secure (14.0% vs. 11.4%, -2.7pp), but the gap is not statistically significant (Fisher's exact p = 0.24, Cohen's h = -0.08) and is a composition artifact of two CWE categories: excluding CWE-502 and CWE-22 eliminates and slightly reverses it (+0.4pp, p = 1.00). A mixed-effects logistic regression confirms sector identity is not a moderator and localizes the only detectable condition effect to those two vulnerability types. 0 of 8 sectors show drift distinguishable from baseline, corrected or uncorrected (|h| < 0.15). A placebo on two non-CISA sectors (e-commerce, online education) reproduces the CISA industry rate almost exactly (10.5% vs. 11.4%, p = 0.63): the small pooled pattern reflects generic industry-framing specificity, not critical-infrastructure identity. In contrast, model selection has a large and consistent effect: among full-output models vulnerability rates range from 11.6% to 16.1%, and these differences persist across conditions. Model choice, not prompt framing, is the more reliable security lever. We release the framework, prompts, generated code, findings, human-validation verdicts, and analysis scripts.

Problem

• security drift
• domain-specific prompting
• AI-generated code
• critical infrastructure
• vulnerability rate

Innovation

• security drift
• sector-conditioned prompting
• LLM code generation
• static analysis
• controlled benchmarking