H-Net++: Hierarchical Dynamic Chunking for Tokenizer-Free Language Modelling in Morphologically-Rich Languages

2025-08-07

AI Summary

To address the high computational cost and lack of morphological awareness in byte-level language modeling for morphologically rich languages, this paper proposes a tokenizer-free, end-to-end dynamic chunking framework. Methodologically, it introduces a hierarchical Transformer architecture integrating a lightweight context mixer, document-level dual-layer implicit priors, orthographic anomaly handling, and progressive curriculum learning—enabling linguistically grounded, adaptive byte chunking. Evaluated on a 1.4-billion-token Persian corpus, the model achieves substantial gains: bit-per-byte (BPB) reduced by 0.159, ParsGLUE accuracy improved by 5.4 percentage points, morphological boundary F1 score of 73.8%, and 53% higher robustness against zero-width non-joiner (ZWNJ) interference. The core contribution lies in the first integration of learnable dynamic chunking with implicit priors—yielding an unsupervised, differentiable, and morphology-aware byte sequence modeling approach.

Technology Category

Natural Language Processing: Lexical Semantics and MorphologyMachine Learning: Large Multimodal Models (LMMs)Computer Vision: Language and Vision

Application Category

Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphsSearch and Retrieval-Augmented AI: Multilingual and cross-lingual Web searchUser Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Byte-level language models eliminate fragile tokenizers but face computational challenges in morphologically-rich languages (MRLs), where words span many bytes. We propose H-NET++, a hierarchical dynamic-chunking model that learns linguistically-informed segmentation through end-to-end training. Key innovations include: (1) a lightweight Transformer context-mixer (1.9M parameters) for cross-chunk attention, (2) a two-level latent hyper-prior for document-level consistency, (3) specialized handling of orthographic artifacts (e.g. Persian ZWNJ), and (4) curriculum-based training with staged sequence lengths. On a 1.4B-token Persian corpus, H-NET++ achieves state-of-the-art results: 0.159 BPB reduction versus BPE-based GPT-2-fa (12% better compression), 5.4pp gain on ParsGLUE, 53% improved robustness to ZWNJ corruption, and 73.8% F1 on gold morphological boundaries. Our learned chunks align with Persian morphology without explicit supervision, demonstrating that hierarchical dynamic chunking provides an effective tokenizer-free solution for MRLs while maintaining computational efficiency.

Problem

• Byte-level models struggle with morphologically-rich languages
• Tokenizer-free segmentation needs linguistically-informed dynamic chunking
• Handling orthographic artifacts in Persian requires specialized solutions

Innovation

• Hierarchical dynamic-chunking model for MRLs
• Lightweight Transformer for cross-chunk attention
• Two-level latent hyper-prior for document consistency