Decoy Images Amplify Caption-Mediated Defenses Against Encoded Jailbreaks

2026-08-02

AI Summary

This study uncovers a critical vulnerability in black-box safety defenses of vision-language models: encoding-based jailbreak attacks suffer a substantial drop in success rates when irrelevant decoy images—such as blank canvases or natural photographs—are introduced. The authors demonstrate for the first time that the mere presence of an image can activate additional safety-check pathways within these defenses, an effect that consistently manifests across diverse models and attack types. Validated through a lightweight encoded-input detector, McNemar’s significance test, and multiple defense frameworks including ECSO, the proposed approach reduces attack success rates by up to 73 percentage points across five mainstream vision-language models while preserving high pass-through rates for legitimate queries, thereby significantly enhancing system robustness.

Abstract

We report a counter-intuitive interaction between image inputs and existing black-box defenses on Vision--Language Models (VLMs): pairing an encoded jailbreak prompt with an unrelated decoy image can sharply lower attack success rate (ASR). The operative change is in the defense pipeline, not in the image. Across five frontier VLMs, two encoded-attack families, and three black-box defenses, a caption-mediated defense (ECSO) that leaves ASR essentially unchanged on text-only encoded input drops it by up to $73$pp once a content-free decoy is attached; every non-saturated contrast is significant under exact McNemar tests. We advance two hypotheses for this pattern, supported by indirect evidence rather than pipeline introspection, since a black-box threat model precludes inspecting vendor internals: caption-mediated defenses branch on image presence, and intrinsic image-side safety engages on image-resident content. Three controls constrain the explanation. Blank-canvas and natural-photograph decoys reproduce the effect on every model, implicating image presence rather than content; the effect replicates on three open-weight VLMs served with no moderation layer, so it is not a vendor-filtering artifact; and a non-symbolic, meaning-based encoder reproduces it, so it is not specific to symbolic obfuscation. Attaching a decoy unconditionally is not deployable --- it raises benign refusal to $20$--$79\%$, an inflation of $+10$ to $+67$pp --- but gating attachment on a lightweight encoded-input detector returns benign refusal to the text baseline while preserving the safety gain wherever the detector fires, making detector recall the binding constraint. Under adaptive attacks that target the caption-mediated re-check, the effect degrades but holds. We frame this as an observation about pipeline interaction, not as a robust defense.

Problem

• encoded jailbreaks
• Vision-Language Models
• black-box defenses
• decoy images
• attack success rate

Innovation

• decoy images
• caption-mediated defense
• encoded jailbreaks
• vision-language models
• black-box defense