EXE-Bench: Ranking the Tradeoffs of AI-based Windows Malware Detectors for Real-World Usability

2026-07-27

AI Summary

This study addresses the lack of a systematic evaluation framework for AI-driven Windows malware detectors, which hinders effective model selection in real-world deployments. To bridge this gap, the authors propose EXE-Bench, a comprehensive benchmark that unifies performance, temporal robustness, adversarial robustness, and computational overhead into a single scoring system. By integrating multidimensional metrics, temporal evolution analysis, content-injection adversarial attacks, and inference resource measurements, EXE-Bench enables fair and holistic model comparison. The evaluation reveals that feature engineering methods grounded in domain knowledge significantly outperform most deep learning models under long-term deployment and adversarial conditions, whereas the latter exhibit superior performance only in initial stages. These findings highlight the limitations of relying solely on post-deployment evaluation and reaffirm the enduring value of expert-driven feature engineering in malware detection.

Abstract

Due to the lack of systematic evaluations, we are not yet able to determine which AI-based Windows malware detector to deploy in production, since existing evaluations (i) differ in terms of data used for both training and testing; (ii) do not consider temporal analysis to showcase whether models withstand the passage of time; (iii) avoid security evaluations with adversarial attacks that could highlight their brittleness against content-injection attacks; and (iv) neglect the computational requirements for deployment, risking slow inference on endpoints. For these reasons, we develop EXE-Bench, a comprehensive benchmark of AI-based Windows malware detectors. EXE-Bench assesses performance, temporal and adversarial robustness, and computational overhead, aggregating them into a single score for direct and fair model comparison. Through EXE-Bench, we highlight how evaluations conducted only after deployment are suboptimal and unable to provide a complete picture of their performance. In particular, through our analysis, we remark how much domain knowledge instilled through feature engineering is still extremely useful in this domain, resisting both time and adversarial attacks, in stark contrast with most of the deep networks that only excel right after deployment.

Problem

• malware detection
• AI-based detectors
• systematic evaluation
• adversarial robustness
• computational overhead

Innovation

• EXE-Bench
• temporal robustness
• adversarial robustness
• feature engineering
• malware detection