Breaking Windows Malware Detection: A Comprehensive Evaluation of Problem-Space Adversarial Robustness

2026-09-28

AI Summary

This study addresses the lack of standardized evaluation for malware detector robustness against problem-space evasion attacks, where sample executability and functionality preservation are frequently overlooked. We establish a unified baseline that preserves executability and conduct a large-scale systematic evaluation involving nine state-of-the-art attacks and eight Windows malware detectors. Our findings reveal that a small set of high-impact transformations outperforms expansive transformation spaces, with merely two complementary attacks sufficing to cover 99% of adversarial samples. Furthermore, raw-byte detectors exhibit pronounced vulnerability, determined jointly by attack type and model representation, while adversarial hardening demonstrates cross-attack failure risks. This work provides critical empirical evidence for the security assessment and defense of malware detection systems.

Abstract

Problem-space evasion attacks have exposed critical weaknesses in machine learning-based malware detectors; yet, their evaluation remains fragmented across models, datasets, and attack methodologies, often neglecting domain-specific requirements such as executability and functionality preservation. We address this gap with a unified, large-scale evaluation of nine state-of-the-art evasion attacks against eight Windows malware detectors, including seven open-source models and one commercial detector, under executability-preserving conditions. Our study analyzes attack effectiveness, complementarity, transferability, and adversarial hardening to evaluate robustness along complementary dimensions. We show that detector vulnerability depends strongly on both model representation and attack type: raw-byte detectors are particularly susceptible to several classes of problem-space manipulation, but no detector family is uniformly robust across all attacks. Importantly, effectiveness is not explained by transformation-space size alone: the strongest attacks can achieve substantially higher success while using fewer distinct transformations and concentrating on a small set of high-impact manipulations. We further show that two complementary attacks are sufficient to cover approximately 99% of the adversarial examples produced by the remaining evaluated attacks. Transferability exhibits a different pattern from direct attack success: attacks with low direct success can produce highly transferable evasions. Finally, adversarial hardening is highly attack- and model-dependent: robustness gains often fail to transfer across attacks and can even increase susceptibility to unseen attacks. These findings highlight limitations in current malware robustness evaluations, establish a comprehensive empirical baseline, and clarify relationships between effectiveness, transferability, and defense robustness.

Problem

• malware detection
• adversarial robustness
• evasion attacks
• problem-space
• transferability

Innovation

• Adversarial Robustness
• Problem-Space Evasion Attacks
• Malware Detection
• Transferability
• Adversarial Hardening