Dual Adversarial Fine-tuning for Enhancing Robustness of Large Vision Language Model

2026-07-21

AI Summary

Large vision-language models exhibit vulnerability to adversarial attacks on visual inputs, and existing defense methods struggle to generalize across diverse tasks. This work proposes a dual adversarial fine-tuning framework that enhances cross-task robustness without modifying the model architecture. By freezing the original vision encoder to extract clean image features as visual supervision and incorporating image-text alignment signals as semantic supervision, the method achieves effective defense simply by replacing the CLIP vision encoder. Notably, it is the first approach to simultaneously improve adversarial robustness across multiple tasks—including zero-shot classification, image captioning, and visual question answering—without requiring task-specific retraining, significantly outperforming current state-of-the-art defenses.

Abstract

While Large Vision-Language Models (LVLMs), represented by LLaVA and GPT-4V, have demonstrated remarkable capabilities, their visual inputs remain vulnerable to adversarial attacks, posing significant security risks. Existing defense methods predominantly target single-task scenarios (e.g., zero-shot classification) and consequently lack generalizability across various multimodal tasks. To address this limitation, we propose a dual adversarial fine-tuning framework that jointly optimizes visual and semantic supervision signals from two modalities, enhancing model robustness while generalizing across multiple downstream tasks. The proposed framework comprises two core components, i.e., $\textbf{Visual}$ supervision branch and $\textbf{Semantic}$ supervision branch. The former branch leverages features from clean images, extracted via a frozen original vision encoder, to guide adversarial robustness while the latter incorporates caption-image alignment as a contextual signal to preserve semantic coherence under attack. Moreover, our method achieves cross-task robustness by simply replacing the CLIP vision encoder in the original model, with no need of separate task-specific retraining or architecture modifications.Extensive experiments demonstrate that our approach outperforms the state-of-the-art method in adversarial robustness evaluation across zero-shot classification, image captioning, and visual question answering (VQA) tasks.

Problem

• adversarial robustness
• large vision-language models
• multimodal tasks
• visual inputs
• security risks

Innovation

• dual adversarial fine-tuning
• visual-language models
• adversarial robustness
• cross-task generalization
• semantic alignment