Lossy Compressive Text Autoencoders

2026-10-07

AI Summary

This study addresses the challenge of efficiently compressing text data while preserving semantic integrity by proposing a discrete bottleneck autoencoder that integrates data compression with representation learning. Methodologically, it introduces a low-dimensional discrete bottleneck combined with a temporal-axis residual scaling architecture to achieve high-fidelity lossy compression of textual latent representations. The approach is comprehensively evaluated using quantization metrics, BLEU scores, and large language models. Experimental results demonstrate that the proposed model attains a compression rate of 2.24 bits per byte on web text datasets, yielding reconstruction quality comparable to lossless algorithms while exhibiting strong performance in downstream tasks. Overall, this work provides an effective solution for simultaneously achieving high compression ratios and semantic preservation.

Abstract

Our work explores learning a compressed latent representation of text, at the intersection of data compression and representation learning. We propose an autoencoder architecture that performs residual downscaling and upscaling of hidden representations along the time axis, with a residual low-dimension discrete bottleneck. We analyze our approach for different quantization methods, training objectives, and datasets. For different levels of compression, we evaluate the similarity between the original and reconstructed text both at the surface-level (BLEU) and at the semantic-level (LLM-based judge). Additionally, we evaluate our models on downstream question-answering and semantic text similarity benchmarks. Our approach results in compressed representations which are on par with lossless text compression algorithms at 2.24 bits per byte on web text data, while having good reconstruction and downstream task performance.

Problem

• lossy text compression
• text autoencoder
• latent representation
• representation learning

Innovation

• Lossy Text Compression
• Autoencoder
• Discrete Bottleneck
• Representation Learning
• Residual Downscaling