LLM-based Source Code Compression via Thresholded Symbol Ranking

2026-07-27

AI Summary

This work addresses the limitations of general-purpose compressors in capturing the intrinsic regularities of source code and the low throughput of existing large language model (LLM)-based compression methods. It proposes a lossless compression approach grounded in a Shannonian symbol-ranking framework: an LLM predicts symbol distributions, and only the top-T symbols (with T=1 or 63) are explicitly encoded, while the remainder are treated as outliers and jointly compressed with the rank stream. The method introduces, for the first time, a threshold-constrained symbol-ranking mechanism that avoids unbounded rank encodings, achieving a better trade-off between compression ratio and throughput. Evaluated across 30 LLMs—including general-purpose, code-specific, and quantized variants—combined with general compressors like zstd, the approach improves compression ratios by up to 37% and speeds up compression by 40% over prior LLM-based methods, and achieves up to 82% better compression than general-purpose compressors, substantially expanding the compression–speed Pareto frontier and revealing that source code is more amenable than natural language to LLM-based modeling of its compressible structure.

Abstract

We study the problem of lossless compression of source code, motivated by the storage demands of large-scale software archives, such as Software Heritage (https://www.softwareheritage.org/). General-purpose compressors (e.g., zstd, bzip2) offer a good trade-off between compression ratio and speed, but fail to exploit all special regularities inherent in source code. Recent approaches leverage Large Language Models (LLMs) within Shannon's symbol-ranking framework, relying on a scheme in which the predicted rank can grow arbitrarily. While effective at reducing space, this setting incurs significant throughput degradation, and leaves open the question whether it is necessary to explicitly encode all ranks. In this work, we introduce LLM-based compressors deploying two novel symbol-ranking variants that bound predictions to the top-$T$ ranks ($T=1$ or $63$), with out-of-threshold symbols stored as exceptions and compressed jointly with the rank stream via general-purpose compressors. We conduct the first large-scale evaluation of LLM-based source code compression across 30 LLMs, including general-domain, code-specialized, and quantized models. Our $T$-bounded approach outperforms prior LLM-based compressors both in compression ratio (up to 37% relative improvement) and compression throughput (40% faster). Compared to general-purpose compressors (e.g., zstd, bzip2), we obtain up to 82% relative compression gain but at a lower speed, thus offering a new trade-off point in the compression-speed spectrum. We also show that these gains are stronger on source code than on natural language, suggesting an interesting indication, namely that source code exposes regularities captured by LLMs but missed by general-purpose exact-match-based compressors. We conclude by commenting on open problems that offer theoretical and practical avenues of research.

Problem

• lossless compression
• source code
• Large Language Models
• symbol ranking
• compression throughput

Innovation

• LLM-based compression
• thresholded symbol ranking
• source code compression
• lossless compression
• compression-throughput trade-off