Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models
This work addresses the limitation of conventional Transformers, which lack a native knowledge retrieval mechanism and rely on inefficient computation to simulate memory access, thereby constraining their performance on knowledge-intensive tasks. The authors propose “conditional memory” as a novel dimension of sparsity, introducing an Engram module based on modern N-gram embeddings to enable O(1) static knowledge lookup. This module is co-optimized with a Mixture-of-Experts (MoE) architecture to balance neural computation and memory invocation. For the first time, scalable static memory is integrated into sparse large language models, revealing a U-shaped scaling law between memory capacity and model performance. The memory module alleviates the burden on the backbone network, allowing it to focus on complex reasoning. The method significantly improves performance on knowledge benchmarks such as MMLU and CMMLU, yields even greater gains on reasoning and code tasks including BBH, ARC, HumanEval, and MATH, and boosts long-context retrieval accuracy from 84.2% to 97.0%.