Language Modeling is Monotone Compression

2026-10-07

AI Summary

This study addresses the unclear theoretical connection between large language models (LLMs) and data compression capabilities. By integrating tools from computational complexity, information theory, and cryptography, this work establishes for the first time a theoretical equivalence between LLMs and monotone compression algorithms, while further deriving the equivalence between next-bit pseudoentropy and monotone incompressibility. Specifically, it proves that LLMs and monotone compression algorithms can be mutually constructed within an additive constant of error, revealing an intrinsic connection between this equivalence and the existence of one-way functions. Moreover, the derived pseudoentropy equivalence constitutes an independent contribution to cryptography, providing a novel theoretical foundation for understanding the computational nature of LLMs.

Abstract

A long-standing hypothesis in artificial intelligence and neuroscience posits that intelligence is closely related to compression: the ability to compress information efficiently intuitively reflects capacities associated with intelligence and learning. Indeed, recent experimental works verify this intuition by showing connections between the capabilities of large language models (LLMs) and their ability as compressors: for instance, Deletang et al. (ICLR'24) demonstrate that LLMs can be used as powerful compressors, and Huang et al. (COLM'24) show that the compression ability of LLMs is highly correlated with their performance on benchmarks for knowledge and reasoning. In this work, we initiate a theoretical study of this connection. Our main result is that LLMs (formally modeled as next-token predictors) are equivalent to monotone (a.k.a. order-preserving) compression algorithms---namely, compression algorithms where the encoding process preserves the ordering of the inputs---in the sense that the one can be constructed from the other while preserving the same error up to an additive gap of 2. We next show that the monotonicity is required for this equivalence to hold if and only if cryptographic (infinitely-often) one-way functions exist. As a direct corollary, we get a cryptographic result of independent interest: the notion of next-bit pseudoentropy (a computational analogue of entropy) of a distribution is equivalent to monotone incompressibility of the distribution. (Previously, it was only known (Haitner et al., ITCS'23) that incompressibility implies next-bit pseudoentropy.)

Problem

• Language Modeling
• Monotone Compression
• Large Language Models
• Next-token Prediction
• One-way Functions

Innovation

• Monotone Compression
• Large Language Models
• Next-token Prediction
• One-way Functions
• Next-bit Pseudoentropy