Splintering Nonconcatenative Languages for Better Tokenization

2025-03-18

AI Summary

Conventional subword tokenization algorithms (e.g., BPE, UnigramLM) assume linear concatenative morphology, making them ill-suited for non-concatenative languages—such as Hebrew and Arabic (root-and-pattern morphology) or agglutinative yet discontinuous languages like Malay—where morphological units are not orthographically contiguous. This leads to poor preservation of morphological integrity during tokenization. Method: We propose SPLINTER, the first preprocessing framework that systematically models non-concatenative morphology’s impact on tokenization. It performs language-aware text linearization by integrating morphological rules with statistical heuristics to reorder morphemic units into contiguous sequences amenable to standard subword algorithms. Contribution/Results: SPLINTER is compatible with both BPE and UnigramLM. Evaluated within the multilingual BERT framework, it improves morphological integrity by +23% on Hebrew, Arabic, and Malay, and yields an average +1.8 percentage point gain on Hebrew downstream tasks—effectively breaking the implicit concatenative-morphology assumption underlying mainstream tokenizers.

Technology Category

Natural Language Processing: Lexical Semantics and MorphologyMachine Learning: Large Multimodal Models (LMMs)Planning, Routing, and Scheduling: Planning with Language Models

Application Category

Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web searchWeb Mining and Content Analysis: Large pretrained models with web dataGraph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

Common subword tokenization algorithms like BPE and UnigramLM assume that text can be split into meaningful units by concatenative measures alone. This is not true for languages such as Hebrew and Arabic, where morphology is encoded in root-template patterns, or Malay and Georgian, where split affixes are common. We present SPLINTER, a pre-processing step which rearranges text into a linear form that better represents such nonconcatenative morphologies, enabling meaningful contiguous segments to be found by the tokenizer. We demonstrate SPLINTER's merit using both intrinsic measures evaluating token vocabularies in Hebrew, Arabic, and Malay; as well as on downstream tasks using BERT-architecture models trained for Hebrew.

Problem

• Addresses tokenization challenges in nonconcatenative languages.
• Proposes SPLINTER for better representation of morphological patterns.
• Improves tokenizer performance in Hebrew, Arabic, and Malay.

Innovation

• SPLINTER pre-processes nonconcatenative language text
• Rearranges text to represent root-template morphologies
• Improves tokenization for Hebrew, Arabic, and Malay