Shared Vulnerabilities in Robustness-Optimized Defenses: One Breach Exposes the Family

2026-07-19

AI Summary

This work reveals that existing robustness-optimizing defenses, while enhancing adversarial robustness, may inadvertently introduce shared intra-family vulnerabilities, such that compromising a single model jeopardizes the entire defense ensemble. For the first time, we demonstrate that such defenses exhibit natural transferability-based weaknesses and propose a strict transfer-only attack protocol alongside a lightweight adaptive attack method, PGDTransfer, which leverages adversarial sensitivity maps (AdvSMs) to quantify vulnerability alignment across different defenses. Experimental results show that under ε = 4/255, PGDTransfer achieves an average transfer attack success rate of 80.4% against mainstream purification-based defenses. Our findings advocate incorporating vulnerability diversity and transfer-only isolation as core security objectives in the design of future robust defenses.

Abstract

Adversarial robustness optimization aims to preserve correct prediction under adversarial perturbations, and has produced substantial robustness gains through methods such as adversarial training and adversarial purification. However, we identify a new security risk: these gains can create shared vulnerabilities across defenses. Once one representative robustness-optimized defense is effectively breached, the broader family may become exposed. Studying this risk requires separating genuine transferability from distortion-induced degradation and from the algorithmic gains of sophisticated attacks. We therefore introduce stricter transfer-only protocols and a deliberately simple adaptive attack, PGDTransfer, to test whether robustness-optimized defenses share transfer-only vulnerability under controlled conditions. We further introduce Adversarial Sensitivity Maps (AdvSMs) to visualize and quantify shared alignment beyond differentiable classifiers, including stochastic and non-differentiable defenses. Across adversarially trained classifiers, purification-based defenses, and LVLMs with robust visual encoders, we identify natural transferability within each robustness family, i.e., transfer that arises even with simple PGD-style optimization rather than specialized transferable-attack design. The risk is already severe for purification: PGDTransfer reaches an average transfer attack success rate of $80.4\%$ across filtering-, compression-, and diffusion-based purifiers under $ε=4/255$, suggesting that purifier defenses may no longer provide reliable protection. As attacks improve, currently stronger robustness families may face the same risk. Future defenses should therefore treat vulnerability diversity and transfer-only isolation as security objectives, rather than optimizing only individual robustness.

Problem

• adversarial robustness
• shared vulnerabilities
• transferability
• robustness-optimized defenses
• adversarial attacks

Innovation

• transferable vulnerability
• adversarial robustness
• PGDTransfer
• Adversarial Sensitivity Maps
• robustness-optimized defenses