EngramEdit: Decoupled Knowledge Updates in LLMs through Conditional Memory

2026-10-07

AI Summary

This study addresses the vulnerability of knowledge editing in large language models to unintended interference with unrelated facts, caused by n-gram activation discrepancies and embedding sharing. To mitigate this, we propose a decoupled editing method based on a conditional memory architecture that transforms conditional memories into editable knowledge interfaces. With the Transformer backbone frozen, our approach computes target memory representations and jointly optimizes shared n-gram embeddings, integrating chain-of-thought prompting to enable independent updates of factual knowledge. Experimental results demonstrate that the proposed method achieves an editing success rate approaching 100% and yields multi-hop reasoning accuracy exceeding three times that of baseline approaches, while effectively preserving unrelated knowledge and general capabilities.

Abstract

Conditional memory architectures such as DeepSeek Engram use input n-grams to look up learned embeddings, expanding the capacity of large language models (LLMs) with limited additional computation. Beyond model scaling, this architecture has demonstrated the potential to decouple factual knowledge storage from general-purpose computation, offering a promising route to updating factual knowledge while keeping the Transformer backbone fixed. Realizing this potential is challenging because different expressions of a fact may activate different n-gram embeddings, while updating shared embeddings can unintentionally change the model's predictions about other facts. We propose EngramEdit for decoupled knowledge updates through conditional memory. EngramEdit first computes target memory representations that make the model predict the updated fact across multiple expressions. It then jointly updates the shared n-gram embeddings to match these targets across expressions and edits, penalizing updates to frequently reused embeddings more strongly to preserve unrelated knowledge. Experiments show that EngramEdit enables independent factual knowledge updates through conditional memory, achieving near-perfect editing success. Revised knowledge is usable across unseen expressions and in multi-hop reasoning, with nearly three times the strongest baseline's accuracy under chain-of-thought (CoT) prompting. Unrelated knowledge and general capabilities are largely preserved even as factual updates accumulate. These findings show that EngramEdit turns conditional memory into an editable knowledge interface, extending its role beyond model scaling to support decoupled knowledge updates.

Problem

• knowledge editing
• conditional memory
• large language models
• decoupled knowledge updates
• n-gram embeddings

Innovation

• Conditional Memory
• Knowledge Editing
• Decoupled Knowledge Updates
• Large Language Models
• N-gram Embeddings