Two Sides of the Same Coin: Co-Evolving Search for Cross-Task Attacks on Vision-Language Models

2026-08-03

AI Summary

This work addresses limitations in existing adversarial attack methods for vision-language models, particularly their inadequate exploration of the search space and poor cross-task transferability. The authors propose a novel cross-modal attack framework that, for the first time, incorporates a co-evolutionary mechanism: it evolves semantic-negative text embeddings on the language side while optimizing region-level image perturbations via population-based strategies on the vision side, jointly steering the source object’s semantics toward a target class. Integrating evolutionary algorithms, momentum-based gradient optimization, and cross-modal adaptive search, the method substantially enhances both attack success rates and transferability. Extensive experiments demonstrate its effectiveness across multiple tasks—including image captioning, object detection, and region classification—on state-of-the-art models such as Florence-2, OFA, and UnifiedIO-2, while ablation studies confirm the complementary and efficient design of its components.

Abstract

Vision-language models (VLMs) exhibit strong generalization across multimodal tasks but remain vulnerable to adversarial perturbations. Existing attacks typically follow single-trajectory gradient optimization or task-specific objectives, limiting search-space exploration and cross-task transferability. We propose an evolutionary-computation-guided cross-modal attack framework for unified VLMs. The framework adaptively searches both textual and visual spaces. On the textual side, it evolves hard negative semantic embeddings around the source-category representation to provide diverse cross-modal repulsion. On the visual side, it maintains a population of object-region perturbations and combines momentum-based gradient updates with evolutionary selection, mutation, and crossover to more reliably explore multiple feasible trajectories. Jointly optimizing semantic negative guidance and localized perturbations generates adversarial examples that consistently shift source-object semantics toward target categories across vision-language tasks. Theoretical analyses show that the co-evolutionary search preserves perturbation feasibility, prevents degradation of the best observed fitness, and increases the probability of reaching high-margin adversarial regions compared with single-trajectory optimization. Experiments on Florence-2, OFA, and UnifiedIO-2 demonstrate strong overall attack performance across image captioning, object detection, region categorization, and object localization. Ablation studies further verify the complementary effectiveness of text-side semantic evolution and image-side perturbation evolution, as well as the framework's efficiency and cross-task transferability.

Problem

• vision-language models
• adversarial attacks
• cross-task transferability
• search-space exploration
• multimodal tasks

Innovation

• co-evolutionary search
• cross-task attack
• vision-language models
• adversarial perturbations
• multimodal optimization