Precision over Scale: A Polish-Silesian Benchmark and a Translation System Outperforming Open-Source and Commercial Models

2026-10-01

AI Summary

This study addresses the dual challenges of data scarcity and the neglect of linguistic variation in standard benchmarks for dialectal machine translation. Focusing on Polish–Silesian translation, this work integrates neural machine translation, rule-based systems, and TranslateGemma fine-tuning techniques, while introducing SiLTT, the first dedicated test set for this language pair. Experimental results on both the SiLTT and BOUQuET benchmarks demonstrate that rule-based systems achieve superior performance; notably, neither fine-tuned large language models nor open-source models surpass these rule-based approaches. The best-performing neural models and evaluation benchmarks have been released as open-source resources. This project provides a critical reference for advancing research in low-resource dialectal translation.

Abstract

Dialectal machine translation remains challenging due to limited data and strong linguistic variation not captured by standard benchmarks, which often assume standardized and well-edited text. We study Polish-Silesian MT using neural and rule-based systems, evaluating on SiLTT - a new Pol-Szl testset, alongside established BOUQuET and FLORES benchmarks. Results show our rule-based system is consistently strongest on SiLTT and BOUQuET datasets and that TranslateGemma fine-tuned on a curated dataset improves over strong neural baselines but does not surpass the rule-based system in dialectal settings. We release SiLTT and our best neural model to support further research.

Problem

• Dialectal machine translation
• Polish-Silesian
• low-resource
• linguistic variation
• benchmark

Innovation

• Dialectal Machine Translation
• Polish-Silesian
• Rule-based System
• Benchmark Dataset
• TranslateGemma Fine-tuning