🤖 AI Summary
Existing prompt injection detectors frequently miss high-severity attacks under distribution shifts while still outputting overconfident predictions near 1.0, a risk substantially underestimated by standard calibration metrics. This work introduces a severity-aware calibration perspective, proposing a severity metric S to quantify the confidence assigned to missed attacks. Through fixed-threshold cross-distribution evaluation, black-box rewrite-based attack generation, and instruction-tuned models as judges, the study systematically investigates detector failure mechanisms. Findings reveal that content keywords—not injection syntax—are the primary cause of detection blind spots. Critically, all evaluated detectors exhibit highly confident false negatives across diverse distribution shifts, exposing fundamental limitations in current calibration approaches.
📝 Abstract
Prompt-injection detectors are deployed as guards: a model scores an input and a downstream system trusts or blocks it on that score. I study the confidence of these scores, not only their accuracy, when the attack distribution shifts away from the clean benchmark on which the operating point was chosen. I evaluate three released detectors, ProtectAI-v2 and two Prompt-Guard-2 checkpoints, at a single source-calibrated threshold that I freeze and transport across five shifts. I report a severity metric S, how confident a detector is on the attacks it misses, alongside the false-negative rate and discrimination. Across every shift and every detector, severity on the missed attacks stays between 0.99 and 1.00 while the false-negative rate ranges from 0.01 to 0.97: when these detectors miss, they miss with near-certainty. All three confidently pass indirect behavior-hijack injection, a blind spot unanimous across two vendors and a fourfold size range. Standard pooled calibration error does not register this; one detector it rates well-calibrated, at 0.06, is miscalibrated at 0.91 on the attacks alone. Run against live models, the missed injections leak the majority of working exploits, passing them at the rate they catch others. A controlled experiment traces the cause to content-keying rather than injection structure, an instruction-tuned model used as a judge shows the same hijack blind spot, and a black-box rewriter exploits the content-keying to manufacture working confident misses, most effectively on the most dangerous attack category. Code and data are public.