TurkBench: A Benchmark for Evaluating Turkish Large Language Models

2026-01-11

arXiv.org

AI Summary

This work addresses the lack of systematic evaluation benchmarks for large language models (LLMs) in non-English languages with unique linguistic characteristics, such as Turkish. To bridge this gap, the authors introduce TurkBench—the first comprehensive evaluation benchmark specifically designed for Turkish—encompassing six core dimensions: knowledge, language understanding, reasoning, content moderation, grammatical and lexical proficiency, and instruction following. TurkBench comprises 21 subtasks and 8,151 expert-curated samples, evaluated under a standardized protocol that supports online submission and automated scoring. By providing a culturally relevant and structurally coherent assessment framework, TurkBench fills a critical void in the multidimensional evaluation of LLMs for Turkish, thereby supporting future research and development in this understudied linguistic domain.

Technology Category

Natural Language Processing: (Large) Language ModelsMachine Learning: Large Multimodal Models (LMMs)Knowledge Representation and Reasoning: Knowledge Representation Languages

Application Category

Search and Retrieval-Augmented AI: Web evaluation methodologies and metricsUser Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendationSemantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

With the recent surge in the development of large language models, the need for comprehensive and language-specific evaluation benchmarks has become critical. While significant progress has been made in evaluating English-language models, benchmarks for other languages, particularly those with unique linguistic characteristics such as Turkish, remain less developed. Our study introduces TurkBench, a comprehensive benchmark designed to assess the capabilities of generative large language models in the Turkish language. TurkBench involves 8,151 data samples across 21 distinct subtasks. These are organized under six main categories of evaluation: Knowledge, Language Understanding, Reasoning, Content Moderation, Turkish Grammar and Vocabulary, and Instruction Following. The diverse range of tasks and the culturally relevant data would provide researchers and developers with a valuable tool for evaluating their models and identifying areas for improvement. We further publish our benchmark for online submissions at https://huggingface.co/turkbench

Problem

• Turkish language
• large language models
• evaluation benchmark
• language-specific evaluation
• generative models

Innovation

• TurkBench
• Turkish language
• large language models
• evaluation benchmark
• multilingual NLP