The Tokens Remember: When Tokenization Bypasses Knowledge Editing and Unlearning

2026-09-24

AI Summary

This study addresses a critical security vulnerability in knowledge editing and unlearning mechanisms for large language models, which overlook tokenizer side channels. Attackers can exploit alternative tokenizations to bypass modifications and recover sensitive information. This work is the first to identify the tokenizer as a security blind spot, proposing Toketive, a reference-free attack framework that leverages tokenization diversity to construct a side channel. By integrating adversarial example generation with probabilistic analysis, Toketive detects tampered content and reconstructs original responses. Experiments across multiple models demonstrate that the proposed method achieves a 38.6% bypass rate, an 84.2% detection F1-score, and a 74.5% reconstruction accuracy, significantly outperforming existing baselines. These findings challenge the conventional assumption regarding the robustness of localized model modifications.

Abstract

Open-weight LLMs give downstream users control over the inference stack, but this flexibility can undermine post-release guarantees that sensitive knowledge has been modified or removed. Model editing and machine unlearning are used to modify or remove targeted knowledge without retraining models from scratch. However, existing security evaluations of these techniques face two critical limitations. First, they typically require access to either the original pre-edit/unlearning model or auxiliary classifiers to detect modifications or reconstruct pre-edit behavior. Second, they evaluate modifications under the canonical tokenization of an input, implicitly treating tokenization as a benign preprocessing step. We show that this assumption creates a security gap: the same input string can be represented by alternative valid tokenizations that induce different computational trajectories, allowing an adversary to bypass localized modifications and recover information intended to be suppressed. We introduce Toketive, a simple yet powerful reference-free attack that exploits the tokenization-based side channel to (i) detect modified knowledge and (ii) reconstruct the corresponding pre-edit response. It operates solely on the released model and requires neither the pre-edit model, training data, shadow models, nor auxiliary classifiers. Across five LLMs, six datasets, and six editing and unlearning techniques, we find that 38.6% of alternative tokenizations bypass the modification and recover the pre-edit response. Toketive detects modified facts with an F1 score of 84.2%, a 26.2% relative gain over the strongest baseline, and reconstructs pre-edit responses with 74.5% top-5 accuracy, 21.7% higher than the best baseline. Our results show that localized modifications should not be treated as robust knowledge-control boundaries without adversarial evaluation over alternative representations.

Problem

• Knowledge Editing
• Machine Unlearning
• Tokenization
• Security Vulnerability
• Large Language Models

Innovation

• Tokenization side channel
• Knowledge editing bypass
• Machine unlearning
• Reference-free attack
• Adversarial evaluation