SectEval: Evaluating the Latent Sectarian Preferences of Large Language Models

2026-03-13

AI Summary

This study addresses implicit biases in large language models (LLMs) when providing information on Sunni and Shia Islam, which may compromise informational fairness. The authors present SectEval, the first systematically constructed bilingual (English/Hindi) benchmark comprising 88 questions, to evaluate response tendencies across 15 prominent LLMs through multilingual prompt engineering, cross-model comparisons, and context-aware experiments. Findings reveal a notable language-dependent bias: models exhibit a pro-Shia inclination in English responses but shift toward Sunni perspectives in Hindi. Intriguingly, some advanced models dynamically adjust their stances based on inferred user geolocation, demonstrating that religious representations in LLM outputs are significantly shaped by both linguistic and geographic contexts. These results underscore the non-neutrality of AI-generated content and raise critical concerns about the objectivity of LLMs in sensitive socioreligious domains.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Philosophy and Ethics of AI: Bias, Fairness & Equity

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

As Large Language Models (LLMs) becomes a popular source for religious knowledge, it is important to know if it treats different groups fairly. This study is the first to measure how LLMs handle the differences between the two main sects of Islam: Sunni and Shia. We present a test called SectEval, available in both English and Hindi, consisting of 88 questions, to check the bias-ness of 15 top LLM models, both proprietary and open-weights. Our results show a major inconsistency based on language. In English, many powerful models DeepSeek-v3 and GPT-4o often favored Shia answers. However, when asked the exact same questions in Hindi, these models switched to favoring Sunni answers. This means a user could get completely different religious advice just by changing languages. We also looked at how models react to location. Advanced models Claude-3.5 changed their answers to match the user's country-giving Shia answers to a user from Iran and Sunni answers to a user from Saudi Arabia. In contrast, smaller models (especially in Hindi) ignored the user's location and stuck to a Sunni viewpoint. These findings show that AI is not neutral; its religious ``truth'' changes depending on the language you speak and the country you claim to be from. The data set is available at https://github.com/secteval/SectEval/

Problem

• Large Language Models
• Religious Bias
• Sectarian Preference
• Sunni
• Shia

Innovation

• SectEval
• religious bias
• large language models
• cross-lingual inconsistency
• geolocation sensitivity