SALT: Salience-Aware Lexical Trie for Long-Context Compression

2026-07-19

AI Summary

This work addresses the high computational and KV cache memory costs incurred by large language models when processing long contexts. Existing compression methods often evaluate sentences in isolation, leading to topic collapse and the loss of infrequent yet critical information. To overcome these limitations, the authors propose SALT, a novel framework that introduces a trie-based structure into long-text compression. SALT leverages sentence frequency as a lightweight proxy for topical importance to construct a keyword trie, which allocates compression budgets according to the document’s thematic structure and enables trie reuse across multi-turn dialogues. By integrating multi-anchor activation with KV caching, SALT substantially reduces prefill overhead while preserving comprehensive topic coverage and remains compatible with existing KV cache optimization techniques.

Abstract

As large language models (LLMs) process increasingly longer prompts, computation and KV-cache memory costs have emerged as major bottlenecks in inference systems. Existing input-level prompt compression methods address this, but rank each sentence by a scalar relevance score, treating the document as an unstructured pool of words and sentences. Under tight budgets, this causes theme collapse, where the dominant theme(s) of a document consumes the budget, discarding less-frequent yet task-relevant themes. Preserving thematic coverage instead requires allocating the budget across recurring themes rather than scoring sentences in isolation. To this end, we propose SALT, a model-agnostic extractive framework that organizes per-sentence keywords into a trie ordered by sentence frequency (SF), a lightweight, reusable proxy for document thematic structure. This trie-based organization smooths memory allocation and prevents dominant themes from monopolizing the budget. Multi-anchor retrieval activates trie nodes labeled by query keywords at any depth, and the trie persists across dialogue turns, supporting multi-turn use without re-encoding the document. By preserving document themes, SALT reduces the prefill computation and memory cost of long-context prompts while remaining composable with KV-cache methods that target decoding-time latency and memory.

Problem

• long-context compression
• theme collapse
• prompt compression
• thematic coverage
• KV-cache memory

Innovation

• lexical trie
• theme-aware compression
• sentence frequency
• multi-anchor retrieval
• KV-cache optimization