Tokenizer-Agnostic Engram Module

2026-07-31

AI Summary

This work addresses the limited reusability of embeddings in the Deepseek Engram module, which stems from its reliance on tokenizer-specific tokenization. To overcome this limitation, the authors propose a tokenizer-agnostic embedding approach that treats N-grams as byte sequences and replaces the original XOR-based hashing with a universal polynomial hash function. This ensures consistent hash representations for byte-equivalent sequences across different tokenizers. By constructing a shared joint embedding space across tokenizers and integrating a conditional memory mechanism, the method substantially enhances the transferability and reusability of the Engram module while maintaining comparable model performance.

Abstract

Deepseek's Engram, a conditional memory module, was introduced to trade-off storage versus reasoning in large language models. However, the module relies on token-level $N$-gram hashing for Engram embedding lookup, introducing a tight coupling to the tokenizer used: a model with a different tokenizer would have to train its own Engram embeddings from scratch. To improve the reusability of Engram embeddings, we propose a change to the hashing routine, enabling compatibility between Engram models using different tokenizers. Instead of modelling disjoint $N$-gram spaces, we treat $N$-gram as a method to sample potentially useful byte sequences, from all possible byte sequences across tokens. We replace the XOR-based hashing with the general polynomial hashing with a joint embedding space across $N$. This work investigates the possible trade-offs and shows that this simple substitution produces comparable performance and achieves tokenizer-agnosticism: hash equivalence for byte-equivalent token sequences.

Problem

• tokenizer-agnostic
• Engram module
• N-gram hashing
• embedding reusability
• large language models

Innovation

• tokenizer-agnostic
• Engram module
• polynomial hashing
• byte-level compatibility
• conditional memory