E-CONAN (Entailment, CONtradition And Neutral) Diagnostics Dataset Investigating Linguistic Phenomena in Arabic Natural Language Understanding

2026-09-27

AI Summary

This study addresses the lack of standardized diagnostic datasets and hierarchical error analysis frameworks for Arabic natural language understanding (NLU). To overcome this bottleneck, the authors construct a cross-lingual NLU error taxonomy, propose an Arabic natural language inference framework, and introduce E-CONAN, a novel diagnostic dataset. The work pioneers macro- and micro-level classification naming conventions and combines manual annotation with comparative experiments involving pretrained models and large language models (LLMs), thereby filling the gap in fine-grained diagnostics for Arabic. Experimental results demonstrate that while LLMs outperform pretrained models in commonsense reasoning, they exhibit comparatively weaker syntactic capabilities. Furthermore, the findings establish inference as the most challenging linguistic phenomenon, offering critical insights for advancing Arabic NLU research.

Abstract

Natural Language Understanding (NLU) plays a crucial role in various applications, yet its performance suffers from weaknesses in handling the complexities of human languages, ranging from lexical ambiguity to high-level reasoning difficulties. Analyzing errors across diverse linguistic phenomena is crucial for NLU improvement, as it will help humans get insights to comprehensively assess models'limitations and capabilities, so optimizing models'generalization. Notably, several benchmarks contain diagnostics datasets designed for investigation and fine-grained error analysis. When highlighting the gaps in the state-of-the-art, we noted that there is no naming convention for macro and micro categories or even a standard set of linguistic phenomena that should be covered. To overcome this gap, we propose an initial hierarchy for Cross-Lingual NLU error analysis. Moreover, we propose a methodology to create an NLI hierarchical framework and applied a case study on Arabic NLU. Moreover, this paper introduces E-CONAN diagnostics dataset, a freely available dataset manually-annotated with coarse-grained and fine-grained categories based on our proposed Arabic hierarchy. E-CONAN dataset helps NLU designers better understand their models by doing error analysis and in-depth investigation. We used E-CONAN to investigate the performance of 9 pretrained language models and 5 LLMs. Results indicate that LLMs outperform pretrained models in world knowledge and commonsense reasoning macro-category, and underperform pretrained models in syntactic macro-category. Moreover, the hardest phenomena for all models is Reasoning, and the easiest phenomena for all pretrained models is Syntactic, and the easiest for LLMs is Lexico-Syntactic.

Problem

• Natural Language Understanding
• Error Analysis
• Diagnostics Dataset
• Arabic NLP
• Linguistic Phenomena

Innovation

• Natural Language Understanding
• Diagnostics Dataset
• Cross-Lingual Error Analysis
• Hierarchical Framework
• Arabic NLI