EntropyMoE: Entropy-Aware Sparse Expert Routing for Tokenizer-Free LLMs

2026-07-31

AI Summary

This work addresses a key limitation in existing tokenizer-free large language models based on dynamic byte chunks: their use of uniform dense computation across all chunks, which fails to account for varying semantic complexity and granularity. To overcome this, we propose EntropyMoE, a novel sparse mixture-of-experts architecture tailored for dynamic byte chunks. EntropyMoE uniquely leverages both chunk entropy and length as joint routing signals to dynamically allocate expert computational resources at the chunk level. This approach successfully extends the mixture-of-experts paradigm to tokenizer-free representations, aligning computational load with semantic complexity. Experimental results demonstrate that EntropyMoE significantly reduces bits-per-byte on held-out data while maintaining competitive downstream task performance, outperforming both dense and sparse baseline models.

Abstract

Recent byte-level large language models (LLMs) have made tokenizer-free modeling increasingly competitive by grouping bytes into dynamically sized patches. However, existing byte-patch architectures still apply the same dense feed-forward computation to every patch. This uniform computation cannot adapt model capacity to variations in patch semantics and granularity. We address this limitation with EntropyMoE, a Mixture-of-Experts (MoE) architecture designed for dynamic byte patches. EntropyMoE replaces the dense feed-forward modules in the global patch Transformer with Top-K expert layers. Each dynamic patch serves as the basic unit of expert routing, and its byte coverage determines its contribution to workload accounting. The router selects experts directly from patch entropy, using the same granularity signal that underlies dynamic patch construction to organize sparse computation. Patch entropy and length jointly define the feature space for regulating expert specialization. Experiments show that EntropyMoE achieves the lowest held-out bits-per-byte among matched dense and sparse baselines while maintaining comparable downstream accuracy. These results establish patch entropy as an effective routing coordinate for sparse conditional computation and extend Mixture-of-Experts modeling beyond tokenizer-based representations.

Problem

• tokenizer-free LLMs
• dynamic byte patches
• sparse computation
• model capacity adaptation
• expert routing

Innovation

• EntropyMoE
• Mixture-of-Experts
• tokenizer-free LLMs
• dynamic byte patches
• entropy-aware routing