CHALIS: A Challenge Dataset for Language Identification in Difficult Scenarios

2026-06-04

AI Summary

Current language identification systems exhibit insufficient robustness when dealing with closely related languages and orthographic noise—such as missing diacritics, cross-script transliteration, homograph character substitution, and internet slang. This work introduces CHALIS, the first systematically constructed challenging benchmark dataset that encompasses mutually intelligible language pairs and incorporates diverse orthographic perturbations through multi-script transliteration, diacritic removal, character substitution, and slang simulation. Evaluation of four state-of-the-art language identification systems on this benchmark reveals significant performance degradation, particularly for low-resource languages and transliterated texts, with markedly increased error rates. These findings expose critical limitations in current approaches and establish a new benchmark and direction for advancing robustness in language identification.

Abstract

We present CHALIS (Challenging Language Identification Samples), a new benchmark dataset explicitly designed to address difficult cases in language identification: cousin languages and orthographic noise. Our dataset has two parts: First, we collected sentences shared across mutually intelligible language pairs (Czech/Slovak, Spanish/Catalan, Portuguese/Galician, Danish/Norwegian). The second part tests for orthography noise: we transliterate text across multiple scripts, remove diacritics, simulate homoglyph attacks, and use Internet slang. We evaluate four widely used language identification systems on CHALIS and demonstrate that all struggle substantially in these scenarios, especially on lower-resource languages within cousin pairs and on transliterated input. The resource is publicly available at https://huggingface.co/datasets/michal-tichy/CHALIS.

Problem

• language identification
• cousin languages
• orthographic noise
• mutually intelligible languages
• transliteration

Innovation

• language identification
• cousin languages
• orthographic noise
• benchmark dataset
• transliteration