Tokenizing Crosslingual Homographs

2026-07-20

AI Summary

This work addresses the challenge that multilingual models often conflate cross-lingual homographs—identical subword forms with distinct meanings or usages across languages—when sharing a unified subword vocabulary. To mitigate this issue, the authors propose a lightweight intervention during tokenization: prefixing shared tokens with language-specific identifier characters to enhance the tokenizer’s sensitivity to linguistic context. This approach is compatible with mainstream tokenization algorithms such as BPE, UnigramLM, and SaGe, and requires no modification to the underlying model architecture. Experimental results demonstrate that the proposed strategy significantly improves the tokenizer’s ability to disambiguate cross-lingual homographs and consistently yields performance gains in English-to-multilingual machine translation tasks.

Abstract

Multilingual language models rely on shared subword vocabularies to represent multiple languages within a limited number of token units. While such sharing is often useful, it can also create cases in which identical surface forms are treated too uniformly across languages, even when their meanings or usage differ. We investigate this limitation through cross-lingual homographs and false friends, and examine whether introducing language information earlier in the tokenization process can improve their treatment. We propose a simple tokenizer-level intervention based on language cues: language-specific characters replacing initial characters of shared-vocabulary words, reducing common identity during vocabulary construction. In intrinsic analysis, we find through tokenizer-level statistics that BPE and UnigramLM often treat cross-lingual homographs in a largely language agnostic way, whereas the context-sensitive SaGe tokenizer diverges more strongly; our intervention removes this gap. In downstream English-to-X machine translation, our cues yield modest improvements in several settings, especially under BPE, although the effect is not consistent across all languages and evaluation sets. Overall, the findings suggest that adding lightweight language information at the tokenizer level is a promising direction for further exploration.

Problem

• cross-lingual homographs
• false friends
• multilingual tokenization
• shared subword vocabulary
• language-specific representation

Innovation

• cross-lingual homographs
• tokenizer-level intervention
• language-specific tokenization
• shared subword vocabulary
• false friends