Evaluating Morphological Plausibility of Subword Tokenization via Statistical Alignment with Morpho-Syntactic Features

2026-01-26

AI Summary

This work proposes a novel metric for evaluating the morphological plausibility of subword segmentations without relying on scarce and inconsistent gold-standard word segmentation data. Instead, it leverages widely available morphosyntactic resources—such as Universal Dependencies and UniMorph—and employs IBM Model 1 to probabilistically align subword units with morphological features. The resulting alignment scores serve as a proxy for morphological well-formedness. The proposed method demonstrates strong correlation with traditional morpheme boundary recall across diverse languages and substantially improves applicability in morphologically complex or low-resource settings, thereby enabling a cross-lingually generalizable framework for morphological evaluation of subword segmentations.

Technology Category

• Natural Language Processing: Lexical Semantics and Morphology
• Machine Learning: Evaluation and Analysis
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Semantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semantics

Abstract

We present a novel metric for the evaluation of the morphological plausibility of subword segmentation. Unlike the typically used morpheme boundary or retrieval F-score, which requires gold segmentation data that is either unavailable or of inconsistent quality across many languages, our approach utilizes morpho-syntactic features. These are available in resources such as Universal Dependencies or UniMorph for a much wider range of languages. The metric works by probabilistically aligning subwords with morphological features through an IBM Model 1. Our experiments show that the metric correlates well with traditional morpheme boundary recall while being more broadly applicable across languages with different morphological systems.

Problem

• subword tokenization
• morphological plausibility
• morpheme boundary
• cross-lingual evaluation
• morpho-syntactic features

Innovation

• subword tokenization
• morphological plausibility
• morpho-syntactic features
• statistical alignment
• IBM Model 1