ROMEVA: Geometry-Preserving Vocabulary Expansion for Roman Urdu Language Models

2026-06-21

AI Summary

This study addresses the severe subword fragmentation in Roman Urdu—averaging 1.50 subwords per token due to spelling variations—which degrades the performance of multilingual language models. To mitigate this, the authors propose ROMEVA, a method that expands mBERT’s vocabulary with 500 high-frequency fragmented tokens and employs subword averaging for initialization alongside a PCA-guided anchor loss to preserve the geometric structure of the embedding space while adapting to the target language. Evaluation on a dataset of 36,130 Roman Urdu comments shows that ROMEVA best retains the pretrained embedding structure; however, standard fine-tuning achieves superior performance on sentiment classification, revealing a trade-off between embedding stability and downstream task effectiveness.

Abstract

Multilingual Language Models like mBERT are widely used for low-resource NLP, yet their adaptation to morphologically inconsistent languages such as Roman Urdu remains underexplored. Roman Urdu spelling variation causes severe sub-word fragmentation, averaging 1.50 sub-words per token. We propose \textit{ROMEVA} (Roman Urdu Embedding-preserving Vocabulary Adaptation), which combines sub-word-average initialization and a PCA-guided anchor loss to stabilize embeddings during vocabulary expansion. Using a 36,130-comment Roman Urdu corpus, we add 500 highly fragmented tokens to mBERT and compare naive fine-tuning, sub-word-aware fine-tuning, and \textit{ROMEVA}. While \textit{ROMEVA} most effectively preserves the pretrained embedding space, naive fine-tuning achieves the strongest downstream sentiment classification performance. These findings reveal a disconnect between embedding stability and downstream performance, suggesting that stronger adaptation may be preferable to strict embedding preservation in morphologically inconsistent languages.

Problem

• Roman Urdu
• sub-word fragmentation
• vocabulary expansion
• multilingual language models
• morphological inconsistency

Innovation

• vocabulary expansion
• embedding preservation
• sub-word fragmentation
• PCA-guided anchor loss
• Roman Urdu