Cartridges at Scale: Training Modular KV Caches over Large Document Collections

2026-06-03

AI Summary

This work addresses the inefficiency of static prefilling in traditional long-context inference and the limited scalability of existing cartridge-based methods to large document collections. The authors propose a scalable, modular framework for KV cache training that enables efficient rotation of thousands of document-level cartridges through dynamic interference mixing, memory budget management, and coordinated scheduling between GPU and persistent storage. This framework achieves, for the first time, composable and scalable joint training across multiple cartridges, overcoming the limitations of monolithic approaches. Integrated with retrieval-guided cartridge selection and high-fidelity compression, the method improves accuracy by 10–31 points over single-cartridge baselines under the same token budget. Compressed oracle performance lags behind full-context learning by only 2–6 points, and when combined with retrieval, it matches or exceeds conventional RAG while reducing prompt token consumption by 3–4×.

Abstract

Large Language Models can reason over long contexts, yet prefilling millions of tokens is wasteful as much of the content remains static across queries. Cartridges address this by distilling document collections into reusable key-value (KV) caches that eliminate prefilling while preserving accuracy. A critical limitation of this approach is that cartridges are monolithic and non-compositional: encoding an entire collection into a single KV block does not scale, and naively mixing cartridges trained in isolation collapses performance to near chance. We introduce Cartridges at Scale (CAS), a training framework for scalable multi-cartridge learning with dynamic distractor mixing and a memory-efficient budget manager that rotates hundreds of per-document cartridges between GPU and persistent storage. Our approach scales to collections exceeding a million tokens, improving over a monolithic cartridge by 10-31 points at comparable token budgets. Oracle cartridge accuracy falls within 2-6 points of full in-context learning even at high compression. When paired with retrieval for cartridge selection, CAS matches or exceeds conventional RAG accuracy while consuming 3-4x fewer prompt tokens.

Problem

• modular KV caches
• large document collections
• scalable inference
• long-context LLMs
• reusable representations

Innovation

• modular KV caches
• scalable training
• dynamic distractor mixing
• memory-efficient budgeting
• retrieval-augmented generation