A Survey of Adversarial Efficiency Degradation for Vision Transformer by Exploiting Input-adaptive Optimization

2026-08-05

AI Summary

This work systematically investigates adversarial efficiency-degradation attacks against input-adaptive inference mechanisms in vision transformers—such as token pruning and early exiting—where adversaries can substantially increase computational overhead with negligible impact on model accuracy. We provide a unified evaluation of two representative attacks, SlowFormer (a universal adversarial patch) and DeSparsify (a single-sample perturbation), across mainstream adaptive frameworks including A-ViT, ATS, and AdaViT. For the first time, we introduce standardized evaluation metrics—GFLOPs, accuracy loss, and attack success rate—and conduct a structured analysis of the attack surface, model vulnerabilities, and existing lightweight defenses. Our experiments reveal the high sensitivity of current adaptive mechanisms to such efficiency-degrading attacks and quantitatively assess both the effectiveness and limitations of proposed defenses in restoring computational efficiency.

Abstract

Vision Transformers (ViTs) increasingly rely on input-adaptive inference, such as token pruning and early halting, to meet energy and latency budgets. This survey examines a recent class of adversarial efficiency degradation attacks that target these mechanisms to increase computation without necessarily degrading accuracy. We unify and compare two representative attacks, SlowFormer (a universal adversarial patch) and DeSparsify (per-image perturbations), across three popular token-pruning frameworks: A-ViT, ATS, and AdaViT. We standardize reporting using GFLOPs, accuracy loss, and an Attack Success (AS) metric that measures how much of the model's compute savings the attack takes away. Understanding these attacks is crucial for designing countermeasures that not only mitigate risk but also remain lightweight, since deployment often occurs in low-power settings such as mobile or embedded devices. To organize our analysis, we focus on three questions: how input-adaptive optimizations (e.g., token pruning and early halting) create attack surfaces for efficiency degradation; how such attacks operate in practice and which optimizations are most vulnerable; and which defenses exist today and whether they meaningfully restore efficiency under attack.

Problem

• adversarial efficiency degradation
• Vision Transformer
• input-adaptive optimization
• token pruning
• early halting

Innovation

• adversarial efficiency degradation
• input-adaptive optimization
• Vision Transformer
• token pruning
• attack success metric