Evaluating the Performance of ChatGPT for Spam Email Detection

2024-02-23

arXiv.org

AI Summary

This study systematically evaluates ChatGPT’s applicability to cross-lingual spam detection, with a focus on low-resource Chinese settings. We employ in-context learning (zero-shot and few-shot) combined with prompt engineering, and benchmark against supervised baselines—including Naïve Bayes, SVM, logistic regression, DNNs, and fine-tuned BERT—on large-scale English and small-scale Chinese datasets. Results show that ChatGPT underperforms supervised models on English data; however, in the low-resource Chinese setting, it surpasses all baselines, achieving up to 92.3% accuracy. To our knowledge, this is the first work demonstrating that large language models can effectively transfer spam detection capability across languages via in-context learning, thereby mitigating data scarcity. Our findings highlight the unique potential and practical utility of LLMs for low-resource language NLP tasks, particularly where labeled training data is severely limited.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Language and Vision

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Large pretrained models with web data
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Email continues to be a pivotal and extensively utilized communication medium within professional and commercial domains. Nonetheless, the prevalence of spam emails poses a significant challenge for users, disrupting their daily routines and diminishing productivity. Consequently, accurately identifying and filtering spam based on content has become crucial for cybersecurity. Recent advancements in natural language processing, particularly with large language models like ChatGPT, have shown remarkable performance in tasks such as question answering and text generation. However, its potential in spam identification remains underexplored. To fill in the gap, this study attempts to evaluate ChatGPT's capabilities for spam identification in both English and Chinese email datasets. We employ ChatGPT for spam email detection using in-context learning, which requires a prompt instruction with (or without) a few demonstrations. We also investigate how the number of demonstrations in the prompt affects the performance of ChatGPT. For comparison, we also implement five popular benchmark methods, including naive Bayes, support vector machines (SVM), logistic regression (LR), feedforward dense neural networks (DNN), and BERT classifiers. Through extensive experiments, the performance of ChatGPT is significantly worse than deep supervised learning methods in the large English dataset, while it presents superior performance on the low-resourced Chinese dataset. This study provides insights into the potential and limitations of ChatGPT for spam identification, highlighting its potential as a viable solution for resource-constrained language domains.

Problem

• Evaluate ChatGPT for spam detection
• Compare with traditional machine learning methods
• Assess performance in English and Chinese datasets

Innovation

• ChatGPT for spam detection
• In-context learning approach
• Comparison with benchmark methods