Investigating Adversarial Robustness of Multi-modal Large Language Models

2026-06-02

AI Summary

This work addresses the vulnerability of multimodal large language models (MLLMs) to adversarial perturbations stemming from their reliance on vision encoders, a weakness inadequately mitigated by existing defenses that strictly align with the original CLIP embedding space. To enhance robustness systematically, the authors propose a three-pronged framework: first, a diagnostic CLIP alignment protocol identifies effective robust vision encoders; second, end-to-end multimodal adversarial training transfers this robustness to the full MLLM; and third, a lightweight random transformation at test time is introduced as a black-box defense mechanism. Experiments demonstrate that the approach improves performance under strong adversarial attacks by an average of 28 CIDEr points and 11.7% VQA accuracy. Moreover, the test-time defense enables non-robust models to approach the performance of robust counterparts and substantially suppresses toxic outputs triggered by visual jailbreaks.

Abstract

Multi-modal Large Language Models (MLLMs) achieve strong performance on vision-language tasks, but incorporating visual inputs through a vision encoder (e.g., CLIP) substantially expands the attack surface, making these models vulnerable to visual adversarial perturbations. Prior defenses typically preserve compatibility with pretrained MLLMs by enforcing strict alignment to CLIP's original embedding space during adversarial fine-tuning; while practical, this constraint fundamentally limits achievable robustness. We present a systematic investigation of adversarial robustness in MLLMs. We first introduce a diagnostic CLIP-alignment protocol that predicts, prior to full MLLM training, which robust vision encoders will transfer effectively to the multimodal setting, revealing that large-scale multimodal adversarial pretraining, rather than unimodal scale alone, is the critical factor for strong robustness transfer. Integrating such encoders into MLLMs via end-to-end multimodal training yields average gains of 28 CIDEr points on captioning and 11.7% VQA accuracy under strong adversarial attacks compared to constrained plug-and-play baselines. We further show that adversarial training applied directly to a standard non-robust MLLM degrades both clean and adversarial performance, establishing robust visual representations as a strict prerequisite, while end-to-end adversarial training from a robust backbone delivers additional gains of 1.9 CIDEr points and 4.3% VQA accuracy. Beyond training-time defenses, lightweight test-time visual stochastic transformations serve as an effective black-box defense for non-robust MLLMs, elevating adversarial performance from near-zero to levels comparable with robust models. Finally, we show that our robust models substantially reduce toxic generation under white-box visual jailbreak attacks. Code and pretrained weights will be released publicly.

Problem

• Adversarial Robustness
• Multi-modal Large Language Models
• Visual Adversarial Perturbations
• Vision Encoder
• Robustness Transfer

Innovation

• adversarial robustness
• multimodal large language models
• vision-language alignment
• end-to-end adversarial training
• test-time defenses