Gradual Forgetting: Logarithmic Compression for Extending Transformer Context Windows

2025-10-24

AI Summary

This work addresses the high computational complexity and memory decay inherent in standard Transformers when processing long contexts. We propose an input-level solution—inspired by human memory cognition—that requires no architectural modifications: a scale-invariant logarithmic compression mechanism maps the original token sequence into a compressed representation whose length grows logarithmically with context size. By operating solely at the input representation layer, our method preserves the standard Transformer architecture intact while substantially extending its effective memory capacity. Language modeling experiments on WikiText-103 and PG-19 demonstrate significant perplexity reduction, with performance consistently improving as the compressed context length increases—confirming enhanced modeling of long-range dependencies. Our key contribution is the first integration of cognition-inspired logarithmic compression into the input representation layer, enabling efficient long-context extension without any structural changes to the model.

Technology Category

Natural Language Processing: (Large) Language ModelsMachine Learning: Learning on the Edge & Model CompressionCognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMsEconomics, Online Markets and Human Computation: Cost models of using LLMs in production systemsGraph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Most approaches to long-context processing increase the complexity of the transformer's internal architecture by integrating mechanisms such as recurrence or auxiliary memory modules. In this work, we introduce an alternative approach that modifies the input representation itself, rather than the transformer architecture. Inspired by cognitive models of human memory, our method applies a scale-invariant logarithmic compression to the input tokens. The resulting compressed representation is processed by a standard, unmodified transformer, preserving architectural simplicity. We evaluate this approach on the WikiText-103 and PG-19 language modeling benchmarks, showing a reduction in perplexity compared to uncompressed baselines. Moreover, performance improves consistently with longer compressed temporal contexts, showing that input-level logarithmic compression is a simple and effective way to extend a transformer's long-range memory.

Problem

• Extends transformer context windows without architectural modifications
• Applies logarithmic compression to input tokens for memory extension
• Improves language modeling performance with longer compressed contexts

Innovation

• Logarithmic compression of input tokens
• Preserves standard transformer architecture
• Extends context window via input modification