Progressive Cramming: Reliable Token Compression and What It Reveals

2026-07-23

AI Summary

This work investigates whether the performance limitations of existing token compression methods stem from optimization failures or fundamental constraints, and questions whether high-fidelity reconstruction alone ensures semantic validity. To this end, the authors propose progressive cramming—a method that incrementally extends the prefix token by token and dynamically identifies compressible boundaries, thereby characterizing the trade-off between compression limits and semantic preservation within a fixed optimization budget in embedding space. The study reveals, for the first time, that compression trajectories exhibit low-dimensional structure in embedding space and demonstrates that perfect reconstruction relies on fragile prompting signals rather than transferable semantics, challenging conventional compression paradigms. Through causal attention ablation, embedding analysis, and multi-benchmark evaluation, experiments show that even when original context is retained, compressed models suffer significant accuracy drops on multiple-choice tasks and nearly lose generative capability, proving that high-fidelity reconstruction does not equate to effective semantic compression.

Abstract

Token cramming compresses sequences into learned embeddings with near-perfect reconstruction, but fixed token budgets and 99\% accuracy thresholds leave it unclear whether residual errors reflect optimization failures or fundamental limits. We introduce progressive cramming, which grows the target prefix token-by-token, stopping only when reconstruction is no longer achievable within a fixed optimization budget. Progressive trajectories occupy low-dimensional structure in embedding space. Prepending a crammed embedding causes a moderate but consistent accuracy drop on multiple-choice benchmarks even with the original prefix in context, and collapses capability almost entirely under generative evaluation. Causal attention-knockout interventions trace this degradation to the embedding's interactions in the model's early layers. These results position progressive cramming as a tool for studying compression limits and show that perfect reconstruction - achievable through brittle steering rather than transferable semantics - is insufficient for meaningful compression.

Problem

• token compression
• reconstruction accuracy
• semantic preservation
• compression limits
• embedding

Innovation

• progressive cramming
• token compression
• embedding reconstruction
• causal attention knockout
• compression limits