CoinRAG: Contextualized Information Nugget KV Cache Reuse for Long-Context RAG

2026-08-07

AI Summary

Traditional RAG systems struggle with information redundancy and noise when processing long contexts, and coarse-grained block-level KV cache reuse fails to simultaneously achieve low prefill latency and high answer accuracy. This work proposes a fine-grained RAG approach that identifies query-relevant semantic units—termed “information nuggets”—through a two-stage retrieval process, then integrates their sliced KV representations with block-level context to construct a compact, semantically focused context representation. The method introduces an offline fine-grained KV cache reuse mechanism, which, under standard fast prefill latency constraints, improves average F1 by 5.3% on LongBench multi-hop question answering tasks while significantly reducing computational overhead, thereby advancing beyond the current Pareto frontier of efficiency and accuracy in RAG systems.

Abstract

Recent optimization studies on Retrieval-Augmented Generation (RAG) have exploited chunk-level KV cache reuse to avoid processing long retrieved contexts for higher efficiency, while significant information redundancy and noise still remain in the coarse-grained chunks. This paper optimizes the Pareto frontier under low prefill latency constraints while maximizing accuracy by proposing CoinRAG (Contextualized Information Nugget KV Cache Reuse for Long-Context RAG). The name metaphorically reflects our core mechanism: much like assembling small tokens (or "coins") to accumulate a larger value, CoinRAG compositionally reuses offline-computed, fine-grained nugget caches to form a learned contextual representation efficiently in a more semantically relevant but compact manner. Specifically, instead of full-chunk encoding, CoinRAG identifies query-relevant semantic units within retrieved chunks through two-stage retrieval and seamlessly assembles their sliced KV representations with a chunk-level context. Extensive evaluations on LongBench multi-hop question answering tasks demonstrate that CoinRAG significantly reduces operational costs and outperforms the other baselines with a new Pareto frontier and an average 5.3% relative improvement in answer quality (F1) under a standard fast prefill latency budget.

Problem

• Retrieval-Augmented Generation
• KV cache reuse
• long-context
• information redundancy
• prefill latency

Innovation

• KV cache reuse
• fine-grained retrieval
• contextualized information nugget
• long-context RAG
• Pareto frontier optimization