Packed Malware Detection Using Grayscale Binary-to-Image Representations

2025-12-17

AI Summary

To address the challenge of effectively detecting packed malware via static analysis, this paper proposes a deep learning approach based on grayscale byte-level images: Windows PE files are converted into fixed-size grayscale images, followed by Gabor feature enhancement and transfer learning using VGG16 or DenseNet121 for binary classification (packed vs. unpacked). This work is the first to systematically demonstrate the strong generalization capability of byte-image representations against unseen packers—achieving cross-packer detection rates exceeding 95%. We further observe complementary performance between the two architectures: DenseNet121 attains the highest accuracy and lowest false positive rate, while VGG16 yields marginally higher recall; both achieve F1-scores above 0.97. The proposed method thus delivers high robustness alongside practical deployability in industrial settings.

Technology Category

Machine Learning: Multi-class/Multi-label Learning & Extreme ClassificationComputer Vision: Representation Learning for VisionNatural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models

Application Category

Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphsWeb Mining and Content Analysis: Large pretrained models with web dataUser Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

Detecting packed executables is a critical step in malware analysis, as packing obscures the original code and complicates static inspection. This study evaluates both classical feature-based methods and deep learning approaches that transform binary executables into visual representations, specifically, grayscale byte plots, and employ convolutional neural networks (CNNs) for automated classification of packed and non-packed binaries. A diverse dataset of benign and malicious Portable Executable (PE) files, packed using various commercial and open-source packers, was curated to capture a broad spectrum of packing transformations and obfuscation techniques. Classical models using handcrafted Gabor jet features achieved intense discrimination at moderate computational cost. In contrast, CNNs based on VGG16 and DenseNet121 significantly outperformed them, achieving high detection performance with well-balanced precision, recall, and F1-scores. DenseNet121 demonstrated slightly higher precision and lower false positive rates, whereas VGG16 achieved marginally higher recall, indicating complementary strengths for practical deployment. Evaluation against unknown packers confirmed robust generalization, demonstrating that grayscale byte-plot representations combined with deep learning provide a useful and reliable approach for early detection of packed malware, enhancing malware analysis pipelines and supporting automated antivirus inspection.

Problem

• Detect packed malware using grayscale binary-to-image representations
• Compare classical feature-based methods with deep learning approaches
• Evaluate generalization on unknown packers for reliable early detection

Innovation

• Grayscale byte-plot representation of binary executables
• CNN models like VGG16 and DenseNet121 for classification
• Robust detection across diverse packers and obfuscation techniques