Multilingual Unlearning in LLMs: Transfer, Dynamics, and Reversibility

2026-06-02

AI Summary

This study addresses the challenge of unlearning sensitive information in large language models within multilingual settings, where existing approaches are predominantly English-centric and lack cross-lingual generalization. The authors extend the TOFU benchmark to five languages and systematically investigate the transferability, mechanisms, and reversibility of multilingual unlearning. Through multilingual fine-tuning, cross-lingual query evaluation, hierarchical representation analysis, and inference-time directional guidance, they find that unlearning effects transfer most strongly between languages sharing linguistic families or scripts, primarily impacting decoder layers rather than shared cross-lingual representations. The work presents the first quantitative characterization of cross-lingual unlearning transfer and demonstrates that up to 50% (Qwen) to 90% (Gemma) of forgotten knowledge can be recovered using a single inference-time directional cue, thereby proposing an effective reversible suppression mechanism.

Abstract

Large language models (LLMs) can memorize sensitive facts, motivating unlearning methods that remove targeted knowledge without costly retraining. However, unlearning research remains heavily English-centric. We study multilingual unlearning by extending the TOFU benchmark to five languages, and fine-tune, unlearn, and query our models with different permutations of languages. We find that unlearning transfer, the ability of an unlearned model to "forget" facts in languages other than the unlearning language, is highly variable: e.g., it is strongest between languages sharing scripts and families, and we show that the unlearning language predicts which query languages are most likely to yield the strongest transfer. Layer-wise analysis reveals that unlearning leaves the shared cross-lingual latent space largely intact in early layers, instead operating primarily in later decoding layers. This suggests that unlearning does not truly erase knowledge, but rather induces superficial suppression. Exploiting this structure, a single inference-time steering direction reverses much of this suppression across languages, recovering 50% (Qwen) and 90% (Gemma) of the unlearned knowledge.

Problem

• multilingual unlearning
• unlearning transfer
• cross-lingual forgetting
• language models
• knowledge erasure

Innovation

• multilingual unlearning
• unlearning transfer
• cross-lingual latent space
• inference-time steering
• language families