Why Pretraining Fails to Share Cross-Lingual Knowledge

2026-09-16

AI Summary

研究解决了大型语言模型跨语言知识迁移有限的问题，通过控制实验发现不同词汇空间是主要障碍，并提出通过简单词级翻译映射到共享词汇空间的方法来改善。

Abstract

Large Language Models (LLMs) have made remarkable progress in the processing and modeling of many languages. Yet, unlike human multilinguals, they exhibit surprisingly limited cross-lingual knowledge transfer. While this limitation is well documented, its origins during multilingual training remain unclear. We pretrain 360M- and 7B-parameter LLMs and show that poor cross-lingual knowledge generalization emerges during pretraining and persists under standard interventions. To isolate its cause, we employ a controlled bilingual pretraining setting using two copies of the same language, sharing identical text and token segmentation, but mapped to disjoint token spaces. We find that disjoint tokens alone are enough to induce knowledge compartmentalization, even between identical copies of the same language, establishing disjoint token spaces as a fundamental barrier to cross-lingual knowledge generalization. Guided by this understanding, we suggest mapping languages into a shared token space by simple word-wise translation and find it substantially improves cross-lingual knowledge generalization, recovering up to 12.6\% of native-language learning efficiency --- 14$\times$ the baseline.

Problem

• Cross-lingual Knowledge Transfer
• Large Language Models
• Multilingual Training

Innovation

• disjoint token spaces
• cross-lingual knowledge generalization
• shared token space