On Success and Simplicity: A Second Look at Transferable Vision-Language Attack Pipeline

2026-07-16

AI Summary

This work addresses the limitations of existing transferable adversarial attack methods for vision-language pretraining models, which often rely on complex loss functions or multi-stage designs, resulting in low efficiency and suboptimal performance. To overcome these issues, we propose Simple Vision-Language Attack (SimVLA), a streamlined single-stage framework that incorporates a domain-knowledge-driven cross-modal keyword identification mechanism. This approach effectively suppresses redundant operations and inappropriate modality interactions, thereby significantly enhancing both attack transferability and computational efficiency. Extensive experiments demonstrate that SimVLA consistently outperforms state-of-the-art methods across four datasets and three downstream tasks. For instance, on Flickr30k, it improves R@1 transferability by 8.01%–14.71% while reducing runtime and GPU memory consumption to approximately 35.73% and 46.26% of the baseline, respectively.

Abstract

Vision-Language Pre-training Models (VLPMs) are known to be vulnerable to adversarial attacks. Recent transferable attacks on VLPMs have followed a common pipeline with complicated loss functions or multi-stage text/image attacks. However, in this paper, we demonstrate that such a sophisticated attack pipeline can be simpler yet more successful. Specifically, we identify three previously overlooked issues caused by inappropriate cross-modal interactions and excessive operations. To address them, we propose the Simple Vision-Language Attack (SimVLA) pipeline, which observably improves transferability and efficiency. Experiments on four datasets and three downstream tasks validate the superiority of our pipeline. For instance, on Flickr30k text-image retrieval dataset, our SimVLA outperforms the SOTA baseline in R@1 transferability by 8.01\%-14.71\%, while consuming only about 35.73\% of the time and 46.26\% of the max VRAM. Overall, the superiority of our SimVLA highlights the importance of leveraging domain knowledge (e.g., our proposed cross-modal word identification), while blindly pursuing intricate operations (e.g, complex loss functions and redundant multi-stage designs) may even be harmful. We hope our SimVLA can serve as a simple yet effective backbone for future extensions. Code is available at https://github.com/RYC-98/SimVLA.

Problem

• Vision-Language Pre-training Models
• Adversarial Attacks
• Transferability
• Cross-modal Interaction
• Attack Pipeline

Innovation

• transferable adversarial attack
• vision-language models
• cross-modal interaction
• adversarial efficiency
• simple attack pipeline