Unifying Adversarially Robust Model Experts in Vision-Language Models

2026-07-30

AI Summary

This work addresses the limited robustness of vision-language models such as CLIP under adversarial attacks and the inability of existing adversarial fine-tuning methods to provide comprehensive defense across diverse attack scenarios. To overcome this, the authors propose CARE, a novel framework that uniquely integrates collaborative multi-expert learning with embedding space alignment. During adversarial fine-tuning, multiple expert models with complementary robustness characteristics are trained, share knowledge through embedding alignment, and are subsequently fused into a single unified model. Experiments demonstrate that this unified model substantially outperforms individual experts on image classification and various downstream vision-language tasks, effectively transcending the limitations of conventional single-strategy fine-tuning and confirming the pivotal role of collaborative learning in enhancing adversarial robustness.

Abstract

Vision-language models (VLMs), such as CLIP, are vulnerable to adversarial attacks, posing a serious problem for real-life applications and deployment. Adversarial fine-tuning emerges as a prominent defense method; however, different fine-tuning strategies often produce specialized models with distinct robustness characteristics. Each fine-tuned model in turn thrives in some evaluation settings but falters on others, limiting their defensive capabilities. We refer to these specialized fine-tuned models as robust model experts and propose a collaborative adversarial fine-tuning framework: CARE - Collaborative Adversarial Robustness fine-tuning using Embedding alignment. CARE maintains multiple experts during training, enables knowledge exchange through embedding-space harmonization, and consolidates the learned knowledge into a single unified robust model. Experts benefit from one another while preserving their individual specializations, enabling the final model to inherit complementary robustness properties. In this paper, we demonstrate CARE on two different adversarial fine-tuning strategies with complementary robustness behaviors. Extensive experiments on classic image classification and downstream vision-language tasks display the effectiveness of our approach, with CARE being able to outperform individually learned model experts. The results suggest that collaborative learning across model experts is a promising direction for improving adversarial robustness.

Problem

• adversarial robustness
• vision-language models
• model experts
• adversarial attacks
• fine-tuning

Innovation

• adversarial robustness
• vision-language models
• collaborative learning
• embedding alignment
• model ensembling