Enabling Low-Resource Language Retrieval: Establishing Baselines for Urdu MS MARCO

2024-12-17

AI Summary

Urdu, a low-resource language, suffers from poor performance and a lack of standardized evaluation benchmarks in information retrieval (IR). Method: We introduce Urdu MS MARCO—the first large-scale, human-verified Urdu IR dataset—constructed via high-quality machine translation of MS MARCO followed by expert linguistic validation. We propose a multilingual IR transfer paradigm tailored to South Asian linguistic characteristics, combining zero-shot cross-lingual transfer with fine-tuning of mT5. Within the mMARCO framework, we train Urdu-mT5, a dedicated Urdu-adapted retrieval model. Results: Urdu-mT5 achieves substantial gains over zero-shot baselines, attaining MRR@10 = 0.247 and Recall@10 = 0.439. This work establishes the first reproducible, rigorously evaluated benchmark for Urdu IR, filling a critical gap in low-resource IR research and providing foundational infrastructure and methodology to advance fairness and equity in multilingual IR.

Technology Category

• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Knowledge Representation Languages

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

As the Information Retrieval (IR) field increasingly recognizes the importance of inclusivity, addressing the needs of low-resource languages remains a significant challenge. This paper introduces the first large-scale Urdu IR dataset, created by translating the MS MARCO dataset through machine translation. We establish baseline results through zero-shot learning for IR in Urdu and subsequently apply the mMARCO multilingual IR methodology to this newly translated dataset. Our findings demonstrate that the fine-tuned model (Urdu-mT5-mMARCO) achieves a Mean Reciprocal Rank (MRR@10) of 0.247 and a Recall@10 of 0.439, representing significant improvements over zero-shot results and showing the potential for expanding IR access for Urdu speakers. By bridging access gaps for speakers of low-resource languages, this work not only advances multilingual IR research but also emphasizes the ethical and societal importance of inclusive IR technologies. This work provides valuable insights into the challenges and solutions for improving language representation and lays the groundwork for future research, especially in South Asian languages, which can benefit from the adaptable methods used in this study.

Problem

• Information Retrieval
• Resource-poor Languages
• Urdu

Innovation

• Urdu Information Retrieval
• Zero-shot Learning
• mMARCO Improvement