grapheme-kit: Grapheme-Level Metrics and Text Processing for Multilingual NLP

2026-07-24

AI Summary

This work addresses the limitations of existing Unicode code point–based text evaluation methods, which often fail to accurately measure character-level errors in complex writing systems where a single grapheme frequently comprises multiple code points. To overcome this, the authors introduce grapheme-kit, an open-source Python library that, for the first time, extends widely used NLP evaluation metrics—such as edit distance and similarity—to the grapheme cluster level. By adhering to Unicode standards for grapheme cluster identification, composition, and decomposition, the proposed approach significantly improves evaluation accuracy for tasks like OCR on scripts with complex orthographies, including Tamil and Sinhala. This advancement provides a precise, grapheme-aware toolkit for text processing in low-resource languages.

Abstract

Existing lexical distance, similarity, and evaluation metrics operate on Unicode code points, which can misrepresent errors in writing systems where a single grapheme is represented by multiple Unicode code points. We introduce grapheme-kit, an open-source Python library that extends these metrics to operate on grapheme clusters instead. The library also provides improved grapheme processing for Tamil and Sinhala, including accurate grapheme cluster identification and grapheme composition/decomposition utilities. Through an OCR case study, we demonstrate that grapheme-level metrics provide a more faithful evaluation of complex scripts.

Problem

• grapheme
• Unicode code points
• multilingual NLP
• lexical metrics
• complex scripts

Innovation

• grapheme-level metrics
• grapheme clusters
• multilingual NLP
• Unicode normalization
• complex scripts