adversarial attack evaluation

Inconsistent and unreliable adversarial robustness evaluations arise from model mismatch, non-verifiable implementations, and unequal computational budgets. To address these issues, this paper introduces AttackBench—a standardized benchmarking framework. AttackBench unifies evaluation using gradient-based attacks, a curated set of standard models, and fully reproducible implementations; it further proposes a novel optimality-based metric and strictly controls experimental conditions to ensure fair comparisons. The framework enables trustworthy ranking of mainstream attack methods, systematically identifies sources of bias in existing evaluations, and significantly improves the reproducibility and credibility of robustness verification. Its modular architecture supports continuous extension and benchmark updates, providing a reliable, open evaluation infrastructure for adversarial robustness research.

Existing adversarial attack evaluations suffer from optimistic bias and irreproducibility due to inconsistent perturbation budgets and non-uniform benchmarks. Method: We propose the first fair and reproducible gradient-based attack evaluation framework, introducing an optimality metric based on multi-attack ensemble estimation, enabling standardized cross-algorithm and cross-model-library assessment under strictly fixed forward/backward query budgets. Contribution/Results: We systematically evaluate over 800 attack configurations on CIFAR-10 and ImageNet, covering more than 100 mainstream implementations. Results reveal that only a few methods exhibit consistent superiority across diverse settings. To foster transparency and rigor, we open-source a benchmarking platform and a dynamic leaderboard—establishing a reliable infrastructure for adversarial robustness research.

Existing black-box adversarial attack research lacks a unified, reproducible evaluation benchmark, hindering progress tracking and rigorous technical analysis. To address this, we introduce the first comprehensive benchmark platform designed specifically for realistic, query-limited, and model-inaccessible scenarios. It systematically integrates 25 query-based and 30 transfer-based attack algorithms, evaluated on CIFAR-10 and an ImageNet subset, with full PyTorch implementation supporting gradient estimation, surrogate model training, feature-space transfer, and query optimization. The platform provides a modular codebase, 14,106 cross-model/dataset evaluations, and deep attribution analysis tools. Experimental results reveal fundamental trade-offs among attack success rate, query complexity, and cross-architecture generalization. Our benchmark establishes a reproducible, extensible standard for robustness evaluation—serving as critical infrastructure for advancing black-box adversarial machine learning research.

This work addresses the limitations of existing adversarial simulation tools, which rely on agent-based instrumentation of target systems, often leaving anomalous artifacts and failing to faithfully replicate human attacker behavior—particularly in critical phases of the cyber kill chain such as initial access and interactive operations. To overcome these shortcomings, the authors propose and implement an open-source attack scripting language coupled with an agentless execution engine that closely emulates real-world attacker tactics. This approach enables high-fidelity, interactive simulation of complete kill chain stages, including initial access, privilege escalation, and lateral movement. Experimental results demonstrate that system logs generated by this method exhibit significantly greater behavioral similarity to those produced by actual human-driven attacks, thereby enhancing the realism and effectiveness of security testing and intrusion detection research.

Existing research on model inversion attacks and defenses lacks standardized, reproducible evaluation benchmarks, leading to unfair method comparisons and unreliable defense assessments. Method: We introduce MIBench—the first comprehensive benchmark for model inversion, integrating 16 state-of-the-art attack and defense algorithms with nine standardized evaluation protocols. It supports modular deployment, multi-dimensional analysis (e.g., input resolution, model architecture, loss functions), and joint evaluation of cross-model/task transferability and robustness–accuracy trade-offs. Contribution/Results: MIBench establishes the first unified, extensible, and operationally grounded evaluation framework for model inversion. It significantly mitigates experimental inconsistency and assessment fragmentation, enabling rigorous, comparable, and reproducible privacy-security evaluations. As a result, it provides a trustworthy, standardized infrastructure for advancing privacy-preserving machine learning research.

This study addresses the critical gap between theory and practice in AI-driven cyberattack prediction, focusing on outdated datasets, limited attack coverage, insufficient model interpretability, weak adversarial robustness, and privacy-ethical risks. Through a systematic review of over 150 benchmark datasets and more than 200 studies, the work introduces a novel multidimensional gap assessment framework based on detection impact, implementation cost, and remediation time to prioritize these challenges. The analysis identifies dataset obsolescence and adversarial robustness as the highest-priority issues, while highlighting interpretability as a cost-effective entry point in resource-constrained settings. Furthermore, the study proposes a tripartite classification of dataset quality—production-ready, research-only, and unusable—alongside a corresponding deployment roadmap, significantly enhancing the practical feasibility and robustness of AI-based cybersecurity systems.

This study addresses the limited and fragmented coverage of existing evaluation benchmarks for attacks on large language models, which impedes comprehensive assessment of defensive capabilities. To bridge this gap, the authors propose the first 4×6 objective-technique matrix grounded in the STRIDE threat modeling framework, synthesizing 507 distinct inference-time attack types extracted from a systematic review of 932 publications. This effort yields a scalable attack taxonomy and a benchmark coverage auditing framework. Through systematic literature synthesis, attack clustering, and benchmark mapping, the analysis reveals that prevailing benchmarks cover at most 25% of the defined threat surface, with several high-severity attack categories entirely absent. The project is open-sourced, providing a repository of 2,521 attack groups to enable ongoing community-driven auditing and evolutionary tracking of evaluation benchmarks.

This study addresses the lack of a systematic evaluation framework for AI-driven Windows malware detectors, which hinders effective model selection in real-world deployments. To bridge this gap, the authors propose EXE-Bench, a comprehensive benchmark that unifies performance, temporal robustness, adversarial robustness, and computational overhead into a single scoring system. By integrating multidimensional metrics, temporal evolution analysis, content-injection adversarial attacks, and inference resource measurements, EXE-Bench enables fair and holistic model comparison. The evaluation reveals that feature engineering methods grounded in domain knowledge significantly outperform most deep learning models under long-term deployment and adversarial conditions, whereas the latter exhibit superior performance only in initial stages. These findings highlight the limitations of relying solely on post-deployment evaluation and reaffirm the enduring value of expert-driven feature engineering in malware detection.

This work addresses the lack of systematic evaluation benchmarks for large language models (LLMs) in security audit log investigation tasks by introducing AuditBench, the first audit log benchmark specifically designed for attack investigation. AuditBench encompasses over 50 real-world scenarios across Linux and Windows systems and focuses on four core tasks: alert classification, persistence mechanism identification, among others. Through multidimensional experiments, the study systematically evaluates the impact of model scale, log representation, prompt design, and fine-tuning strategies on performance and error patterns, while also analyzing the quality of LLM-generated explanations. The findings reveal the capability boundaries and characteristic failure modes of various models across different investigative tasks, providing empirical foundations for deploying and optimizing LLMs in security operations.

This study addresses the vulnerability of machine learning–based malware detection systems to adversarial attacks by constructing the first large-scale, real-world adversarial malware dataset with fine-grained labels at both family and type levels, comprising 77,943 PE adversarial samples generated using diverse techniques. Leveraging EMBER feature extraction, VirusTotal metadata analysis, and model retraining experiments, the work quantitatively evaluates the effectiveness of both evasion and poisoning attacks. Results demonstrate that adversarial samples achieve evasion rates of 98.35% (at the family level) and 92.20% (at the type level) against EMBER-based detectors. Moreover, injecting merely 0.5% poisoning samples during retraining elevates the evasion rate from 26.1% to 92.8%, starkly exposing critical robustness deficiencies in current models.