GeoThreat: Transferable Targeted Adversarial Attacks on Large Vision-Language Models for Remote Sensing Image Interpretation

2026-07-23

AI Summary

This work addresses the challenge of achieving transferable, targeted semantic manipulation in remote sensing image interpretation with large vision-language models, which requires balancing local discriminative cues and global scene context. To this end, we propose GeoThreat, a novel transferable targeted adversarial attack method tailored for remote sensing imagery. GeoThreat innovatively integrates conceptual representations (class tokens) with perceptual representations (image patch tokens weighted by cooperative importance estimation), aligning target semantics through cross-attention mechanisms. It further refines local cues and semantic associations via gradient optimization based on adversarial-target similarity and employs an ensemble strategy to iteratively generate perturbations. Extensive experiments demonstrate that GeoThreat significantly outperforms existing methods across multiple large vision-language models, achieving notable advances in both attack transferability and controllability.

Abstract

Adversarial attacks against large vision-language models (LVLMs) serve as an effective means of assessing their robustness in cross-modal semantic understanding. Existing studies mainly focus on corrupting visual inputs to induce predefined erroneous responses in general vision-language tasks, whereas corresponding investigations in remote sensing fields remain largely underexplored. Compared with natural image understanding, remote sensing image interpretation requires joint reasoning over local discriminative cues and global scene context. This poses additional challenges to achieving transferable semantic manipulation toward specified responses under black-box settings. To tackle these challenges, we propose GeoThreat, a transferable targeted adversarial attack method against LVLMs for remote sensing image interpretation. Specifically, GeoThreat modulates adversarial representations in accordance with the target content at both conceptual and perceptual levels. The class tokens from surrogate image encoders are employed as conceptual representations, while perceptual representations are distilled from patch tokens of the adversarial example through collaborative importance estimation. Beyond merely rolling out attention scores across layers, we incorporate adversarial-target similarity gradients to more faithfully characterize the relevance of local visual cues to the intended semantic manipulation. The perceptual representations are then dynamically aligned with target patch tokens in a cross-attentive manner, facilitating the adaptation of local cues toward designated semantic details. Finally, adversarial perturbations are iteratively updated via ensemble-based joint optimization of conceptual calibration and perceptual adaptation. Extensive experiments across diverse LVLMs demonstrate the superiority of GeoThreat in both transferability and controllability.

Problem

• adversarial attacks
• large vision-language models
• remote sensing image interpretation
• transferable targeted attacks
• black-box settings

Innovation

• transferable adversarial attack
• vision-language models
• remote sensing image interpretation
• conceptual-perceptual alignment
• targeted semantic manipulation