Inspect India Evals: An Open Benchmarking Framework for Evaluating Large Language Models in the Indian Linguistic and Cultural Context

2026-07-28

AI Summary

This study addresses the limitations of current mainstream large language model (LLM) evaluation benchmarks, which are predominantly English- and Western-centric and thus inadequately assess model safety, fairness, and accuracy in India’s multilingual and multicultural context. Building upon the UK AISI’s Inspect AI platform, the authors introduce the first open-source evaluation framework tailored to India’s 22 official languages. The framework encompasses six dimensions: multilingual MMLU, localized bias testing (BharatBBQ), multi-turn jailbreak resistance, cultural knowledge assessment, and safety related to digital public infrastructure (DPI), alongside an LLM-as-judge automated scoring mechanism. Evaluations of five open-source models (8B–32B parameters) reveal that Sarvam-M 24B and Gemma 2 27B both achieve 80% on an Indian fairness index, with Sarvam-M excelling in cultural knowledge and DPI compliance. While all models uniformly reject harmful multilingual prompts (100% refusal rate), their DPI safety scores vary widely (20%–100%).

Abstract

India is a vast nation of over 1.4 billion people, varied by hundreds of diverse and locally specific traditions and cultures and 22 officially recognized languages. Large language models (LLMs) are now being deployed on a massive scale throughout the mainland as well as in remote villages. However, the common benchmarks - MMLU, BIG-Bench, and TruthfulQA are almost exclusively English- and Western-centric. They do not identify those safety, fairness, and accuracy failures unique to the Indian context. That is the gap Inspect India Evals seeks to fill. It is an open-source framework built on top of UK AISI's Inspect AI platform. It has six benchmarks: Multilingual MMLU across sixteen Indian languages, BharatBBQ (our adaptation of BBQ for Indian social bias), a safety evaluation for Digital Public Infrastructure, a multilingual safety test using harmful prompts in Indian languages, a multi-turn jailbreak resistance test, and an Indian cultural knowledge benchmark scored using LLM-as-judge rubrics. In this study, we tested five open-weight models ranging from 8B to 32B parameters. Sarvam-M 24B and Gemma 2 27B came out on top, both scoring 80% on the composite India Fairness Index, with Sarvam-M even beating larger 32B models on Indian cultural knowledge and DPI safety compliance. All models scored 100% refusal on Multilingual Safety, whereas DPI safety varied from 20% to 100%. The framework is public. It's built to work with the UK AISI registry. Anyone can reproduce or extend this work.

Problem

• large language models
• India
• multilingual evaluation
• cultural context
• benchmarking

Innovation

• multilingual evaluation
• cultural contextualization
• safety benchmarking
• Indian language LLMs
• open-source framework