🤖 AI Summary
This work addresses the longstanding scarcity of high-quality monolingual encoders for low-resource languages such as Latvian by systematically developing and evaluating Latvian-specific pretrained language models based on RoBERTa, DeBERTaV3, and ModernBERT architectures. These models are optimized using large-scale monolingual corpora and support long-context processing. The best-performing model, lv-deberta-base (111 million parameters), substantially outperforms existing monolingual and multilingual baselines across multiple Latvian diagnostic and linguistic benchmarks, effectively bridging the gap in high-quality pretrained resources for the language. All models and evaluation materials are publicly released to support further research and development in Latvian natural language processing.
📝 Abstract
Encoder-only transformers remain essential for practical NLP tasks. While recent advances in multilingual models have improved cross-lingual capabilities, low-resource languages such as Latvian remain underrepresented in pretraining corpora, and few monolingual Latvian encoders currently exist. We address this gap by pretraining a suite of Latvian-specific encoders based on RoBERTa, DeBERTaV3, and ModernBERT architectures, including long-context variants, and evaluating them across a diverse set of Latvian diagnostic and linguistic benchmarks. Our models are competitive with existing monolingual and multilingual encoders while benefiting from recent architectural and efficiency advances. Our best model, lv-deberta-base (111M parameters), achieves the strongest overall performance, outperforming larger multilingual baselines and prior Latvian-specific encoders. We release all pretrained models and evaluation resources to support further research and practical applications in Latvian NLP.