Linguistic Neuron Overlap Patterns to Facilitate Cross-lingual Transfer on Low-resource Languages

2025-08-23

AI Summary

Low-resource languages exhibit poor zero-shot cross-lingual performance in large language models (LLMs), and fine-tuning is prohibitively expensive. To address this, we propose BridgeX-ICL—a tuning-free, data-efficient method grounded in “language bridging.” First, leveraging the MUSE bilingual dictionary, we construct neuron-probing datasets and introduce the Hilbert–Schmidt Independence Criterion (HSIC) to quantify internal language phylogenetic relationships within LLMs, enabling principled selection of optimal bridge languages. Second, we identify cross-lingually shared neuron subsets to enhance knowledge transfer during in-context learning. Evaluated on two cross-lingual tasks across 15 language pairs spanning seven language families, BridgeX-ICL significantly improves zero-shot performance for low-resource languages. Moreover, our analysis uncovers intrinsic neural overlap mechanisms underlying multilingual representations in LLMs—revealing that shared activation patterns across languages reflect structural linguistic relatedness rather than mere surface-level similarity.

Technology Category

• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Language and Vision

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

The current Large Language Models (LLMs) face significant challenges in improving performance on low-resource languages and urgently need data-efficient methods without costly fine-tuning. From the perspective of language-bridge, we propose BridgeX-ICL, a simple yet effective method to improve zero-shot Cross-lingual In-Context Learning (X-ICL) for low-resource languages. Unlike existing works focusing on language-specific neurons, BridgeX-ICL explores whether sharing neurons can improve cross-lingual performance in LLMs or not. We construct neuron probe data from the ground-truth MUSE bilingual dictionaries, and define a subset of language overlap neurons accordingly, to ensure full activation of these anchored neurons. Subsequently, we propose an HSIC-based metric to quantify LLMs' internal linguistic spectrum based on overlap neurons, which guides optimal bridge selection. The experiments conducted on 2 cross-lingual tasks and 15 language pairs from 7 diverse families (covering both high-low and moderate-low pairs) validate the effectiveness of BridgeX-ICL and offer empirical insights into the underlying multilingual mechanisms of LLMs.

Problem

• Improving zero-shot cross-lingual learning for low-resource languages
• Exploring shared neuron activation patterns across languages
• Enhancing performance without costly fine-tuning methods

Innovation

• BridgeX-ICL method for cross-lingual transfer
• Neuron overlap patterns from bilingual dictionaries
• HSIC-based metric for optimal bridge selection