🤖 AI Summary
This work addresses the high computational and storage costs, methodological fragmentation, and lack of a unified theoretical foundation in memory management for large language model agents. Framing multi-level memory compression through rate-distortion theory, the study formulates it as an optimization problem of preserving task-critical information under resource constraints. It introduces a general compression objective and a layer-agnostic lower bound, establishes a seven-dimensional taxonomy, and achieves, for the first time, cross-layer transfer of compression mechanisms. The analysis reveals fundamental limitations of attention magnitude and temporal decay as universal retention signals. Furthermore, the authors develop a unified evaluation benchmark with reference experiments, distill design principles for memory compression in multi-turn interactions, and systematically outline open challenges in the field.
📝 Abstract
Large language models, and the agents built on them, spend an ever-growing share of their compute and memory on remembering: caching attention keys and values, carrying long prompts, maintaining recurrent state, and storing what happened in previous turns and sessions. Because none of this memory is free, four largely separate research communities have each learned to compact it. They evict or quantize the KV cache, prune or distill prompts, bound architectural state, and consolidate agent memory. We argue that these are instances of one problem: a rate--distortion decision about what context-derived information to retain versus discard, at what fidelity, under a resource budget, so as to preserve downstream task utility. We make this lens precise with a single compaction objective and a layer-agnostic lower bound, use it to build a seven-axis taxonomy that classifies methods from across the stack uniformly, and use it to transfer mechanisms between layers that have never been connected, from serving-stack KV management to agent long-term memory. Two patterns hold across the survey. At every layer the signal that decides what to keep is attention magnitude or recency, and it fails in the same way everywhere, by discarding, before the query is known and with no way to undo it, information the query later needs. And while compression is measured carefully on single-turn long context, the repeated compaction that agents actually perform is almost never measured, and no benchmark holds one budget axis across all the layers at once. We turn both observations into a benchmark proposal, a small reference experiment, and a set of compaction-aware design principles, and we map the open problems.