Visual-Invariance-Augmented Feature Optimal Alignment for Transferable Adversarial Attacks against Closed-Source MLLMs

2026-10-03

AI Summary

This study addresses the limited transferability in black-box transfer attacks against closed-source multimodal large models, where global feature alignment often neglects local structures. To overcome this limitation, we propose IAU-FOA, a method that incorporates visual invariance augmentation—specifically pixel rescaling and white balancing—to simulate illumination variations and enhance robustness. Furthermore, it introduces a confidence-adaptive unbalanced optimal transport mechanism to optimize weakly matched regions, achieving fine-grained alignment between global semantics and local patch features. Extensive experiments demonstrate that IAU-FOA significantly outperforms existing state-of-the-art methods on both open-source and closed-source models, effectively improving the cross-model transfer success rate of adversarial examples.

Abstract

Multimodal large language models (MLLMs) remain vulnerable to transferable adversarial examples, especially in black-box settings where only open-source surrogate models are accessible. Existing targeted transfer attacks mainly align adversarial and target samples using global image-level features, such as encoder [CLS] embeddings. However, such coarse alignment insufficiently exploits patch-level visual structures, limiting transferability across heterogeneous closed-source MLLMs. We propose IAU-FOA, a visual-invariance-augmented feature optimal alignment attack with adaptive unbalanced transport, to improve targeted transferability against closed-source MLLMs. IAU-FOA aligns adversarial and target samples at both global and local levels: a cosine-based objective narrows their global semantic gap, while patch tokens are clustered into compact local patterns and matched through optimal transport for fine-grained feature alignment. Balanced optimal transport enforces fixed marginal masses even for local clusters without reliable counterparts, potentially introducing misleading alignment gradients. We therefore introduce confidence-adaptive unbalanced transport to relax these constraints for weakly matched clusters, aiming to reduce unreliable local alignment and improve adversarial transferability. We further study the effect of input transformations and propose visual-invariance augmentation, which applies bidirectional pixel-intensity rescaling and per-channel white-balance adjustment to simulate exposure, contrast, illumination, and color-temperature variations. This strategy encourages adversarial perturbations to generalize across different visual encoders. Extensive experiments on open-source and closed-source MLLMs show that IAU-FOA consistently outperforms state-of-the-art transferable attack methods. Code is available at https://github.com/jiaxiaojunQAQ/IAU-FOA.

Problem

• transferable adversarial attacks
• multimodal large language models
• black-box setting
• feature alignment
• adversarial transferability

Innovation

• Transferable Adversarial Attacks
• Multimodal Large Language Models
• Optimal Transport
• Feature Alignment
• Visual Invariance