Which LoRA? An Empirical Study on the Effectiveness of LoRA Techniques During Multilingual Instruction Tuning

2026-06-09

AI Summary

This study investigates whether existing LoRA variants offer advantages over standard LoRA in balancing cross-lingual transfer and knowledge retention during multilingual instruction tuning. We systematically evaluate the base LoRA and four of its variants on two multilingual datasets, complemented by an analysis of hidden embeddings to compare internal language representations. Our empirical results—presented for the first time—demonstrate that more complex architectural modifications to LoRA do not yield significant improvements in cross-lingual adaptability. The variants perform comparably to standard LoRA in both cross-lingual transfer and knowledge retention, and fine-tuned models exhibit highly similar inter-layer language representations across all variants. These findings challenge the prevailing assumption that enhanced LoRA architectures inherently confer superior multilingual capabilities.

Abstract

We investigate whether commonly available LoRA variants have an advantage over basic LoRA in multilingual instruction tuning. Experiments involving LoRA and four other variants on two datasets across diverse target languages show that there is no significant advantage in using more complex LoRA variants instead of basic LoRA, with respect to balancing cross-lingual transfer and knowledge retention. An analysis of hidden embeddings reveal that layer-wise language representation remains largely similar across LLMs fine-tuned with different LoRA techniques, suggesting that architectural novelty of LoRA techniques may not translate into better cross-lingual adaptation.

Problem

• LoRA
• multilingual instruction tuning
• cross-lingual transfer
• knowledge retention
• language representation

Innovation

• LoRA
• multilingual instruction tuning
• cross-lingual transfer
• parameter-efficient fine-tuning
• empirical study