ArabicDialectSafety: A Dialect-Aware Benchmark for Arabic Content Safety Classification

2026-08-02

AI Summary

This study addresses the lack of fine-grained benchmarks covering diverse Arabic dialects for evaluating content safety in language models. The authors construct a safety dataset comprising 25,071 human-annotated prompts across six Arabic dialects and introduce the first dialect-aware dual-task evaluation framework that jointly performs binary safety detection and fine-grained harm classification. Experimental results demonstrate that incorporating dialect information at the representation layer yields the best performance; fine-tuned MARBERTv2 achieves Macro-F1 scores of 0.95 and 0.90 on the two tasks, substantially outperforming prompt-based large language models, which exhibit unsafe response rates below 5% on harmful dialectal prompts. The findings reveal a notable performance gap in state-of-the-art models when handling low-resource Maghrebi dialects.

Abstract

We present ArabicDialectSafety, a human-curated Arabic safety dataset of 25,071 prompts covering six Arabic varieties: Modern Standard Arabic, Syrian, Egyptian, Algerian, Palestinian, and Moroccan. The dataset is annotated with dialect labels and seven fine-grained harm categories. We introduce a dual-task evaluation framework for binary safe/unsafe detection and granular harm classification across dialects. Benchmarking seven supervised and generative models, we find that fine-tuned MARBERTv2 achieves the strongest performance, with Macro-F1 scores of 0.95 for binary classification and 0.90 for granular classification, substantially outperforming prompted frontier LLMs, including Arabic-specialized models. Our analyses show that dialect conditioning is most effective when integrated at the representation level, while significant performance gaps remain for low-resource Maghrebi dialects. We further evaluate seven frontier LLMs as response generators on harmful dialectal Arabic prompts and observe unsafe generation rates below 5 percent across models. We release the dataset and code upon acceptance to support future research on dialect-aware Arabic safety evaluation. Warning: This paper contains examples of harmful and potentially offensive content included solely for research purposes.

Problem

• Arabic dialects
• content safety
• harm classification
• dialect-aware evaluation
• low-resource dialects

Innovation

• dialect-aware safety
• Arabic content moderation
• fine-grained harm classification
• representation-level dialect conditioning
• low-resource dialect evaluation