Breaking the Curse ofMultilinguality inMany-to-Many Speech-to-Text Translation via a Resource-AwareMixture of Speech Encoders

2026-08-05

AI Summary

This work addresses the performance degradation of low-resource languages in multilingual speech translation caused by representation competition within shared encoders. To mitigate this issue, the authors propose the MSRT framework, which introduces a resource-aware Mixture-of-Speech-Encoders (MoSE) coupled with an explicit language router. This architecture dynamically routes input speech to either frozen experts—preserving high-resource language capabilities—or trainable experts dedicated to mid- and low-resource languages. A five-stage curriculum learning strategy further reduces data dependency. Evaluated across 45 languages and 1,980 translation directions, the method achieves state-of-the-art performance, effectively aligning representations with as little as 10 hours of paired data per language. Crucially, it significantly improves translation quality for low-resource languages without compromising high-resource performance, thereby overcoming the multilingual curse.

Abstract

Multimodal large language models (MLLMs) have achieved significant success in speech-to-text translation (S2TT). However, when processing multilingual speech inputs, a single speech encoder shared across all languages suffers from the curse of multilinguality: languages at different resource levels compete for limited representation capacity, leading to strong high-resource performance but substantial degradation on low-resource speech. To address this problem and improve multilingual consistency, we propose MSRT, a novel framework built around a resource-aware Mixture of Speech Encoders (MoSE). MoSE uses an explicit language router to assign each utterance to an appropriate expert encoder. A frozen expert preserves high-resource language capabilities, while a trainable expert adapts to and specializes in medium- and low-resource languages. We further introduce a five-stage curriculum learning strategy that substantially reduces data dependence, requiring only 10 hours of paired S2TT data per language for effective alignment. We conduct extensive experiments on 45 languages, systematically evaluating all $45 \times 44$ translation directions. Our 4B-parameter model achieves state-of-the-art performance, outperforming substantially larger baselines. Empirical analyses show that MoSE improves high-, medium-, and low-resource languages simultaneously, with the largest gains on low-resource speech, thereby breaking the curse of multilinguality without compromising high-resource performance. To support future multilingual S2TT research, we release our code and models.

Problem

• multilinguality
• speech-to-text translation
• low-resource languages
• representation capacity
• language competition

Innovation

• Mixture of Speech Encoders
• Resource-Aware Routing
• Curriculum Learning
• Multilingual Speech-to-Text Translation
• Low-Resource Adaptation