HalluTruthQA: A Fine-Grained Benchmark for Hallucination Detection, Localization, and Explanation in Arabic Question Answering

2026-07-22

AI Summary

This work addresses the challenge of factuality hallucinations in large language models (LLMs) for Arabic question answering, which are often difficult to detect and explain due to the absence of fine-grained benchmarks. The authors introduce the first fine-grained hallucination evaluation benchmark for Arabic, encompassing four tasks—detection, localization, verification, and explanation—with 2,400 expert-annotated samples spanning Islamic knowledge, history, science, and geography. Key innovations include character-level hallucination annotations, human-generated explanations, a multi-candidate answer verification mechanism, and a macro/micro hallucination taxonomy. Using expert crowdsourcing and zero-shot evaluation, the study systematically assesses models such as Allam, Falcon-H1, Qwen32, and Silma with multidimensional metrics including F1-Sp, LO-Score, Macro-F1, and explanation scores. Results reveal substantial room for improvement, with best-reported performance at Macro-F1 = 0.880 (detection), F1-Sp = 0.516 (localization), LO-Score = 0.852 (verification), and explanation score = 0.644.

Abstract

Large language models (LLMs) can generate fluent Arabic answers, yet factual errors remain difficult to detect, localize, explain, and verify. Existing hallucination benchmarks often provide response-level labels, with limited support for identifying the exact erroneous content, explaining why it is incorrect, or selecting the correct factual answer. We introduce \textsc{HalluTruthQA}, a fine-grained benchmark for hallucination evaluation in Arabic question answering. The benchmark contains 2,400 expert-curated examples across four knowledge-intensive domains: Islamic knowledge, history, science, and geography. Each example pairs an Arabic question and a model-generated answer with a verified reference answer, a binary hallucination label, six candidate answers for factual verification, and, for hallucinated answers, character-level erroneous spans, human-written explanations, and macro and micro hallucination types. We evaluate four open-source LLMs, \textsc{Allam}, \textsc{Falcon-H1}, \textsc{Qwen32}, and \textsc{Silma}, in a zero-shot setting across hallucination detection, span-level localization, factual verification, and explanation evaluation. Results show that these tasks capture different abilities: no single model achieves the strongest performance across all tasks, with best scores of 0.880 Macro-F1 for detection, 0.516 F1-Sp for localization, 0.852 LO-Score for factual verification, and 0.644 final score for explanation evaluation. Our taxonomy shows that hallucination evaluation should move beyond detection toward localizing, verifying, and explaining factual errors. The code, dataset, prompts, and evaluation scripts are available at https://gitlab.com/nlpresearcher/HalluTruthQA.

Problem

• hallucination
• Arabic question answering
• fine-grained evaluation
• fact verification
• error explanation

Innovation

• hallucination detection
• fine-grained benchmark
• span-level localization
• factual verification
• explanation generation