Language Models are not Equally Robust to Non-Canonical Tokenization across Languages

2026-07-29

AI Summary

This study addresses the lack of systematic analysis on the robustness of language models to non-canonical tokenization across languages and their cross-lingual consistency. Evaluating prominent large language models—Llama-3.1-8B, Qwen3-8B, and Gemma-3-12B—across 27 languages under non-canonical tokenization, we reveal for the first time that tokenization robustness is not a universal model property but highly dependent on language-specific characteristics and their interaction with the tokenizer. Instruction-tuned models exhibit average performance drops of 9.9%–23.7%, with languages exhibiting higher tokenization fragmentation showing greater sensitivity. To mitigate this, we propose a LoRA-based fine-tuning strategy leveraging multi-tokenization sampling in English, which significantly enhances cross-lingual robustness to tokenization variation.

Abstract

Despite the existence of exponentially many valid tokenizations for a given string, language models operate on a single canonical sequence deterministically produced by the tokenizer, leaving the broader tokenization space largely uncharacterized. In this paper, we investigate this overlooked space by studying the behavior of language models under non-canonical tokenizations across diverse languages. For English, prior work shows that models are largely invariant to alternative tokenizations that represent the same underlying string. We ask whether this invariance generalizes to other languages beyond English. We conduct a multilingual study across 27 languages spanning diverse scripts and evaluate LLM behavior under alternative tokenizations across six downstream tasks. We find that tokenization invariance does not generalize: model behavior varies substantially across languages with instruction-tuned models exhibiting an average relative performance drop of 23.7% for Llama-3.1-8B, 11.4% for Qwen3-8B, and 9.9% for Gemma-3-12B. The variation of tokenization invariance is systematic across languages. Languages that exhibit higher token fragmentation show significantly greater sensitivity to non-canonical tokenizations. Our study of tokenization robustness serves as a diagnostic of how tightly a model is coupled to its tokenizer. These results demonstrate that tokenization robustness is not a universal property of language models, but depends strongly on the language and its interaction with the tokenizer. We also show that LoRA fine-tuning with multi-tokenization training data provides an effective mitigation for tokenization sensitivity. Fine-tuning on English alone improves tokenization robustness across languages, while systematically sampling diverse non-canonical tokenizations achieves the strongest overall performance.

Problem

• tokenization robustness
• non-canonical tokenization
• multilingual language models
• tokenizer-model coupling
• token fragmentation

Innovation

• tokenization robustness
• multilingual evaluation
• non-canonical tokenization
• LoRA fine-tuning
• tokenizer-model coupling