E-CONAN (Entailment, CONtradition And Neutral) Diagnostics Dataset Investigating Linguistic Phenomena in Arabic Natural Language Understanding
This study addresses the lack of standardized diagnostic datasets and hierarchical error analysis frameworks for Arabic natural language understanding (NLU). To overcome this bottleneck, the authors construct a cross-lingual NLU error taxonomy, propose an Arabic natural language inference framework, and introduce E-CONAN, a novel diagnostic dataset. The work pioneers macro- and micro-level classification naming conventions and combines manual annotation with comparative experiments involving pretrained models and large language models (LLMs), thereby filling the gap in fine-grained diagnostics for Arabic. Experimental results demonstrate that while LLMs outperform pretrained models in commonsense reasoning, they exhibit comparatively weaker syntactic capabilities. Furthermore, the findings establish inference as the most challenging linguistic phenomenon, offering critical insights for advancing Arabic NLU research.