Structured Memory for Edge Language Models: Persistent Context and Corpus Retrieval via O(1) SSM State Injection

2026-08-03

AI Summary

This work addresses the high latency of conventional Retrieval-Augmented Generation (RAG) on edge devices, caused by context prefilling and KV cache overhead, which impedes real-time interaction. The authors propose PRECOG, a mechanism integrating Structured Memory Consolidation (SMC) with state space models (SSMs), leveraging their fixed-size hidden states to encode document corpora offline and inject the optimal matching state directly at query time. This approach achieves O(1) complexity for zero-context reinjection and enables hierarchical persistent memory fusion, circumventing the positional entanglement limitations inherent in Transformers. Evaluated on a 1.2B-parameter TENNs-LLM, the method reduces prefill latency from 27 seconds to under 6 milliseconds—a speedup of approximately 4,500×—enabling real-time edge interaction while preserving answer quality comparable to traditional RAG.

Abstract

Retrieval-augmented generation (RAG) imposes a prefill cost proportional to retrieved context length, and -- with Transformer backbones -- a KV-cache that grows with each generated token. State-Space Models (SSMs) avoid the second cost by construction; we eliminate the first, collapsing prefill from $O(L_{context})$ to $O(1)$ per query. We introduce PRECOG (Pre-Computed Context Injection), a retrieval mechanism that exploits a property unique to SSMs: the fixed-size, position-agnostic recurrent hidden state is a complete summary of everything the model has read. PRECOG pre-encodes document corpora offline as SSM hidden states and injects the best-matching state directly at query time, bypassing in-context re-ingestion entirely. The same state-injection mechanism enables SMC (Structured Memory Consolidation): a hierarchical persistent memory with cognitive-domain clustering, an adjustable fidelity-vs-storage dial, and $O(1)$ session initialization, which consolidates short-term episodic states into long-term semantic memory and fuses both with retrieved corpus states at query time. We demonstrate the system on TENNs-LLM, a 1.2B-parameter gated-SSM language model with a 192 KB hidden state. PRECOG matches in-context RAG answer quality, reducing prefill latency from $\sim$27 s to $<$6 ms on edge hardware -- a $\sim$4500$\times$ speedup that crosses the threshold from unusable to interactive. The mechanism is architecturally impossible for Transformer KV-caches, which are position-entangled and grow linearly with context length.

Problem

• Retrieval-Augmented Generation
• Prefill Latency
• KV-Cache
• Edge Language Models
• Context Length

Innovation

• State-Space Models
• Retrieval-Augmented Generation
• O(1) Context Injection
• Structured Memory
• Edge Language Models