Robustness of Selected Learning Models under Label-Flipping Attack

2025-01-21

AI Summary

This study systematically evaluates the robustness of ten mainstream machine learning models—SVM, Random Forest, GBM, XGBoost, LightGBM, Gaussian Naïve Bayes (GNB), Multilayer Perceptron (MLP), CNN, MobileNet, and DenseNet—under label-flipping attacks, using a malware detection benchmark. Method: We implement all models uniformly via scikit-learn, PyTorch, and TensorFlow; construct controlled noisy training sets with varying label corruption ratios; and conduct repeated experiments with statistical significance testing to ensure result reliability. Contribution/Results: To our knowledge, this is the first large-scale, cross-model comparison of robustness under label noise. Results show that MLP achieves superior robustness: maintaining ≥82% accuracy post-attack despite high clean-data accuracy (95%), outperforming all others. Tree-based models exhibit heightened sensitivity even to low corruption rates, while lightweight CNN variants demonstrate consistently poor robustness. Our findings reveal MLP’s optimal trade-off between predictive accuracy and resilience to label perturbations, providing empirical guidance for model selection in security-critical applications.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Computer Vision: Adversarial Attacks & Robustness
• Natural Language Processing: Safety and Robustness

Application Category

• Security and Privacy: Security and privacy of machine learning and AI applications
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

In this paper we compare traditional machine learning and deep learning models trained on a malware dataset when subjected to adversarial attack based on label-flipping. Specifically, we investigate the robustness of Support Vector Machines (SVM), Random Forest, Gaussian Naive Bayes (GNB), Gradient Boosting Machine (GBM), LightGBM, XGBoost, Multilayer Perceptron (MLP), Convolutional Neural Network (CNN), MobileNet, and DenseNet models when facing varying percentages of misleading labels. We empirically assess the the accuracy of each of these models under such an adversarial attack on the training data. This research aims to provide insights into which models are inherently more robust, in the sense of being better able to resist intentional disruptions to the training data. We find wide variation in the robustness of the models tested to adversarial attack, with our MLP model achieving the best combination of initial accuracy and robustness.

Problem

• Learning Models
• Malicious Label Modification
• Stability under Attack

Innovation

• Robustness Analysis
• MLP Model
• Malicious Label Alteration