Rate-Utility Frontiers for Language Encodings: Comparing Tokens, Bytes, and Pixels Under Controlled Linguistic Content

2026-07-17

AI Summary

This study addresses the challenge of fairly comparing the information-preserving capabilities of prevalent language encoding schemes—such as subwords, bytes, and pixels—whose evaluation is often confounded by architectural differences. Within a unified framework, the authors employ a controlled bottleneck architecture and parallel corpora spanning 13 languages and 5 scripts to systematically disentangle the effects of input length, encoding capacity, and task-relevant information, thereby mapping rate–utility frontiers. Innovatively integrating three utility metrics—surface-form preservation, cross-lingual alignment, and topic classification—the work reveals that encoding choice fundamentally entails a trade-off among task objective, language-script combination, and computational budget: pixels best preserve surface forms, bytes excel in intra-script alignment, and subwords are most effective for topic prediction. Crucially, performance disparities cannot be attributed solely to sequence length; shorter inputs risk semantic loss, whereas longer inputs retain compressible yet informative content.

Abstract

Language models encode text as subword tokens, raw bytes, or rendered pixels, but these encodings are usually compared under modeling constraints that expose different amounts of linguistic content to models across different languages. We instead ask what each encoding preserves when both the content and the downstream capacity are controlled. Using verified parallel sentences across thirteen languages and five scripts, we compare tokens, bytes, and pixels through a shared bottleneck whose width is swept to trace rate-utility frontiers. This separates three quantities that are often conflated: the number of input positions an encoding creates, the latent capacity available after encoding, and the task-relevant information that survives compression. We evaluate three utilities: surface form preservation, cross-lingual sentence alignment, and topic classification. No encoding dominates across tasks or capacity regimes. Pixels preserve surface form best, bytes preserve cross-lingual alignment best, especially in same-script multilingual settings, and tokens support topic prediction best. These performances are not explained by sequence length alone. Short inputs can discard useful meaning, while long inputs can preserve information that compresses well. Choosing an encoding is therefore not a fixed preference for tokens, bytes, or pixels, but a rate-utility tradeoff that depends on the task, language mix, capacity regime, and compute budget.

Problem

• language encodings
• rate-utility tradeoff
• subword tokens
• bytes
• pixels

Innovation

• rate-utility frontier
• text encoding
• cross-lingual alignment
• information preservation
• bottleneck analysis