GhostPrompt: Cross-Image Adversarial Prompt for Vision-Language Models

2026-07-21

AI Summary

Existing text-based adversarial attack methods typically optimize prompts for individual images, limiting their transferability across different images and thus hindering generalization when targeting vision-language models. To address this, this work proposes GhostPrompt, which introduces—for the first time—the concept of cross-image transferable adversarial prompts. By employing a joint optimization strategy that alternately constructs worst-case visual conditions across diverse images and updates a shared prompt, GhostPrompt distills image-invariant adversarial features into the prompt. This approach establishes an efficient “optimize-once, apply-to-many” attack paradigm, achieving over 30% higher attack success rates on mainstream vision-language models compared to state-of-the-art methods while reducing computational overhead by approximately 70%.

Abstract

Vision-Language Models (VLMs) are known to be vulnerable to adversarial attacks, where subtle perturbations to images or texts induce erroneous outputs. However, most text-based attacks are adapted from language-model-centric methods, in which the visual input is fixed during optimization, resulting in adversarial prompts that are tied to specific images and thus limiting their attack effectiveness. To this end, we first introduce a new research perspective: cross-image transferability for adversarial prompts. We then propose GhostPrompt, an adversarial prompt that is optimized once and reused to steer VLM outputs toward attacker-specified responses across diverse images. GhostPrompt employs a joint optimization that distills image-invariant adversarial features into the prompt by "worst-case" generation. Specifically, it alternates between constructing hard visual conditions for the current prompt and updating the prompt to remain effective under these conditions. Extensive experiments on prevalent VLMs verify that \ourmethod achieves an improvement of over 30% in attack success rates compared to state-of-the-art (SoTA) baselines, while reducing computation time by ~70%. Our code is avalable at https://github.com/Ye-ze-yu/GhostPrompt.

Problem

• adversarial prompt
• cross-image transferability
• vision-language models
• text-based attacks

Innovation

• cross-image transferability
• adversarial prompt
• vision-language models
• joint optimization
• image-invariant features