Investigating Retrieval-Augmented Generation in Quranic Studies: A Study of 13 Open-Source Large Language Models

2025-03-20

International Journal of Advanced Computer Science and Applications

AI Summary

Large language models (LLMs) exhibit religious hallucinations and deviations from authoritative Qur’anic texts in Quranic studies. Method: We propose the first multi-granular, structured Retrieval-Augmented Generation (RAG) system tailored to all 114 Surahs, integrating semantic, historical, and textual feature knowledge. Contribution/Results: Through systematic evaluation of 13 open-source LLMs on religious QA tasks, we demonstrate that a RAG-enhanced small model (Llama3.2-3B) achieves faithfulness (4.619) and relevance (4.857) scores comparable to larger models—challenging the conventional trade-off between parameter count and reliability. We introduce a human-annotated, three-dimensional evaluation framework—assessing context relevance, answer faithfulness, and answer relevance—and empirically confirm that RAG substantially mitigates hallucination, thereby enhancing the trustworthiness of LLMs in religious domains.

Technology Category

• Natural Language Processing: Question Answering
• Machine Learning: Large Multimodal Models (LMMs)
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Accurate and contextually faithful responses are critical when applying large language models (LLMs) to sensitive and domain-specific tasks, such as answering queries related to quranic studies. General-purpose LLMs often struggle with hallucinations, where generated responses deviate from authoritative sources, raising concerns about their reliability in religious contexts. This challenge highlights the need for systems that can integrate domain-specific knowledge while maintaining response accuracy, relevance, and faithfulness. In this study, we investigate 13 open-source LLMs categorized into large (e.g., Llama3:70b, Gemma2:27b, QwQ:32b), medium (e.g., Gemma2:9b, Llama3:8b), and small (e.g., Llama3.2:3b, Phi3:3.8b). A Retrieval-Augmented Generation (RAG) is used to make up for the problems that come with using separate models. This research utilizes a descriptive dataset of Quranic surahs including the meanings, historical context, and qualities of the 114 surahs, allowing the model to gather relevant knowledge before responding. The models are evaluated using three key metrics set by human evaluators: context relevance, answer faithfulness, and answer relevance. The findings reveal that large models consistently outperform smaller models in capturing query semantics and producing accurate, contextually grounded responses. The Llama3.2:3b model, even though it is considered small, does very well on faithfulness (4.619) and relevance (4.857), showing the promise of smaller architectures that have been well optimized. This article examines the trade-offs between model size, computational efficiency, and response quality while using LLMs in domain-specific applications.

Problem

• Addressing hallucinations in LLMs for Quranic query responses
• Evaluating 13 open-source LLMs with RAG for domain-specific accuracy
• Balancing model size, efficiency, and quality in religious contexts

Innovation

• Retrieval-Augmented Generation enhances Quranic studies accuracy
• Evaluates 13 open-source LLMs across different size categories
• Utilizes descriptive Quranic dataset for contextually faithful responses