Fine-Tuning LLMs for Low-Resource Dialect Translation: The Case of Lebanese

2025-04-30

AI Summary

This work addresses the trade-off between cultural authenticity and data scale in low-resource Lebanese Arabic machine translation, proposing a novel paradigm: “cultural adaptation over data volume.” Methodologically, it introduces a joint contrastive fine-tuning and contrastive prompting strategy applied to the open-source Aya23 model, incorporating base fine-tuning, contrastive fine-tuning, and syntactic prompting. It also establishes LebEval—the first native-speaker–curated evaluation benchmark for Lebanese Arabic. Key contributions include the first empirical validation that small-scale, culturally authentic, locally sourced data (LW) substantially outperforms large-scale non-local data: LW yields a +12.7 BLEU improvement on LebEval. Results demonstrate that optimizing translation for low-resource dialects should prioritize contextual and cultural fidelity over indiscriminate expansion of training data volume.

Technology Category

• Natural Language Processing: Lexical Semantics and Morphology
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Large pretrained models with web data
• Security and Privacy: Large-scale security measurements

Abstract

This paper examines the effectiveness of Large Language Models (LLMs) in translating the low-resource Lebanese dialect, focusing on the impact of culturally authentic data versus larger translated datasets. We compare three fine-tuning approaches: Basic, contrastive, and grammar-hint tuning, using open-source Aya23 models. Experiments reveal that models fine-tuned on a smaller but culturally aware Lebanese dataset (LW) consistently outperform those trained on larger, non-native data. The best results were achieved through contrastive fine-tuning paired with contrastive prompting, which indicates the benefits of exposing translation models to bad examples. In addition, to ensure authentic evaluation, we introduce LebEval, a new benchmark derived from native Lebanese content, and compare it to the existing FLoRes benchmark. Our findings challenge the"More Data is Better"paradigm and emphasize the crucial role of cultural authenticity in dialectal translation. We made our datasets and code available on Github.

Problem

• Effectiveness of LLMs in low-resource Lebanese dialect translation
• Impact of culturally authentic data versus larger translated datasets
• Challenging the 'More Data is Better' paradigm in dialect translation

Innovation

• Contrastive fine-tuning with bad examples
• Culturally authentic small dataset usage
• LebEval benchmark for native evaluation