SeDeM: Selective Decompression of Hidden-State Memories for Long-Context Question Answering

2026-07-31

AI Summary

This work addresses the challenges of high inference cost and large KV cache memory consumption in long-context question answering with large language models, where existing compression methods struggle to balance information retention and decoding efficiency. The authors propose SeDeM, a novel framework that decouples memory storage from decoding conditions for the first time: hidden states extracted from intermediate Transformer layers are lightly compressed and stored, and a learnable query-guided selector retrieves relevant memory chunks via a chunk-level evidence supervision mechanism. Only the selected chunks are selectively decompressed for decoding. This approach avoids processing the full context or generating answers directly from highly compressed memory. Evaluated on four long-context QA benchmarks, SeDeM outperforms existing compression methods, surpasses full-context fine-tuning with a 3B-parameter model, and simultaneously reduces first-token latency while improving autoregressive decoding throughput.

Abstract

Long-context inference with large language models (LLMs) is costly: self-attention during prefill scales quadratically with sequence length, and the key-value (KV) cache grows with the number of processed tokens. Larger context windows also do not ensure reliable evidence use. Context compression reduces this cost, but many soft-compression methods use LLMs as compressors and rely on compact memory tokens both to preserve information and to condition the decoder. We propose SeDeM, a selective decompression framework that decouples compact memory storage from decoder conditioning. An LLM extracts hidden states from a chosen intermediate Transformer layer, a lightweight compressor stores them as memory blocks, a query-conditioned selector selects relevant blocks, and a decompressor expands only the selected blocks into hidden states compatible with an intermediate decoder layer. Thus, the decoder avoids both full-context processing and direct generation from highly compressed memory slots. On four long-context QA benchmarks, SeDeM achieves higher QA scores than the evaluated compression baselines in both 1B and 3B same-backbone settings, and with the 3B backbone exceeds full-context fine-tuning on three datasets. The learned selector uses block-level evidence supervision during training. SeDeM also reduces online time-to-first-token and improves autoregressive decoding throughput relative to ICAE.

Problem

• long-context question answering
• context compression
• KV cache
• hidden-state memory
• evidence utilization

Innovation

• selective decompression
• hidden-state memory
• context compression
• query-conditioned selection
• long-context QA