Connecting the Persian-speaking World through Transliteration

2025-02-27

AI Summary

Tajik speakers, who use the Cyrillic script, face significant barriers in accessing Persian-language web content written in the Perso-Arabic script (e.g., Iranian/Farsi and Afghan Dari texts). Although formal registers across these Persian dialects are highly similar, parallel corpora are scarce. To address this, we propose orthographic transliteration—not machine translation—as a more practical solution. We introduce the first bilingual Tajik–Farsi transliteration dataset covering both script directions. We design a character-level Transformer-based grapheme-to-phoneme (G2P) model tailored to Persian dialect phonology and incorporate context-sensitive alignment to resolve ambiguous character mappings. Experiments reveal pronounced bidirectional asymmetry in transliteration performance (chrF++: 58.70 vs. 74.20), reflecting inherent script-structural disparities. We publicly release both the dataset and model, establishing the first benchmark for Persian-script transliteration.

Technology Category

• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Machine Learning: Large Multimodal Models (LMMs)
• Application Domains: Humanities & Computational Social Science

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Despite speaking mutually intelligible varieties of the same language, speakers of Tajik Persian, written in a modified Cyrillic alphabet, cannot read Iranian and Afghan texts written in the Perso-Arabic script. As the vast majority of Persian text on the Internet is written in Perso-Arabic, monolingual Tajik speakers are unable to interface with the Internet in any meaningful way. Due to overwhelming similarity between the formal registers of these dialects and the scarcity of Tajik-Farsi parallel data, machine transliteration has been proposed as more a practical and appropriate solution than machine translation. This paper presents a transformer-based G2P approach to Tajik-Farsi transliteration, achieving chrF++ scores of 58.70 (Farsi to Tajik) and 74.20 (Tajik to Farsi) on novel digraphic datasets, setting a comparable baseline metric for future work. Our results also demonstrate the non-trivial difficulty of this task in both directions. We also provide an overview of the differences between the two scripts and the challenges they present, so as to aid future efforts in Tajik-Farsi transliteration.

Problem

• Transliterate Tajik Persian to Perso-Arabic script
• Enable Tajik speakers to access Persian internet content
• Address challenges in Tajik-Farsi script differences

Innovation

• Transformer-based G2P approach
• Novel digraphic datasets used
• Achieved high chrF++ scores