BlackboxBench: A Comprehensive Benchmark of Black-box Adversarial Attacks

2023-12-28

arXiv.org

AI Summary

Existing black-box adversarial attack research lacks a unified, reproducible evaluation benchmark, hindering progress tracking and rigorous technical analysis. To address this, we introduce the first comprehensive benchmark platform designed specifically for realistic, query-limited, and model-inaccessible scenarios. It systematically integrates 25 query-based and 30 transfer-based attack algorithms, evaluated on CIFAR-10 and an ImageNet subset, with full PyTorch implementation supporting gradient estimation, surrogate model training, feature-space transfer, and query optimization. The platform provides a modular codebase, 14,106 cross-model/dataset evaluations, and deep attribution analysis tools. Experimental results reveal fundamental trade-offs among attack success rate, query complexity, and cross-architecture generalization. Our benchmark establishes a reproducible, extensible standard for robustness evaluation—serving as critical infrastructure for advancing black-box adversarial machine learning research.

Abstract

Adversarial examples are well-known tools to evaluate the vulnerability of deep neural networks (DNNs). Although lots of adversarial attack algorithms have been developed, it is still challenging in the practical scenario that the model's parameters and architectures are inaccessible to the attacker/evaluator, i.e., black-box adversarial attacks. Due to the practical importance, there has been rapid progress from recent algorithms, reflected by the quick increase in attack success rate and the quick decrease in query numbers to the target model. However, there is a lack of thorough evaluations and comparisons among these algorithms, causing difficulties of tracking the real progress, analyzing advantages and disadvantages of different technical routes, as well as designing future development roadmap of this field. Thus, in this work, we aim at building a comprehensive benchmark of black-box adversarial attacks, called BlackboxBench. It mainly provides: 1) a unified, extensible and modular-based codebase, implementing 25 query-based attack algorithms and 30 transfer-based attack algorithms; 2) comprehensive evaluations: we evaluate the implemented algorithms against several mainstreaming model architectures on 2 widely used datasets (CIFAR-10 and a subset of ImageNet), leading to 14,106 evaluations in total; 3) thorough analysis and new insights, as well analytical tools. The website and source codes of BlackboxBench are available at https://blackboxbench.github.io/ and https://github.com/SCLBD/BlackboxBench/, respectively.

Problem

• Evaluating black-box adversarial attack algorithms comprehensively
• Comparing attack success rates and query efficiency
• Providing modular codebase and analytical tools

Innovation

• Unified modular codebase for 59 attack algorithms
• Comprehensive evaluations on 2 datasets and models
• Provides analytical tools and new insights