MTP-S2UT: Enhancing Speech-to-Speech Translation Quality with Multi-token Prediction

2025-10-11

AI Summary

In speech-to-speech translation (S2ST), speech tokens often exhibit sparse semantics, and individual tokens struggle to represent complete semantic units. To address this, we propose Hidden-layer Multi-Target Prediction (MTP): within the speech-to-unit translation (S2UT) framework, we apply multi-target prediction loss not only at the output layer but also to intermediate hidden layers, thereby encouraging the model to learn denser and more holistic semantic representations earlier in the encoding process. This design is jointly optimized with Connectionist Temporal Classification (CTC) loss to achieve hierarchical semantic enhancement. Experiments across multiple benchmarks demonstrate that all MTP variants consistently improve translation quality; notably, MTP-S2UT achieves state-of-the-art performance, validating that strengthening semantic representation in hidden layers significantly enhances the model’s capacity for speech translation.

Technology Category

• Natural Language Processing: Speech
• Machine Learning: Mixture of Experts (MoE)
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Current direct speech-to-speech translation methods predominantly employ speech tokens as intermediate representations. However, a single speech token is not dense in semantics, so we generally need multiple tokens to express a complete semantic unit. To address this limitation, we introduce multi-token prediction (MTP) loss into speech-to-unit translation (S2UT) models, enabling models to predict multiple subsequent tokens at each position, thereby capturing more complete semantics and enhancing information density per position. Initial MTP implementations apply the loss at the final layer, which improves output representation but initiates information enrichment too late. We hypothesize that advancing the information enrichment process to intermediate layers can achieve earlier and more effective enhancement of hidden representation. Consequently, we propose MTP-S2UT loss, applying MTP loss to hidden representation where CTC loss is computed. Experiments demonstrate that all MTP loss variants consistently improve the quality of S2UT translation, with MTP-S2UT achieving the best performance.

Problem

• Enhancing semantic density in speech-to-speech translation
• Addressing late-stage information enrichment in translation models
• Improving hidden representation quality through multi-token prediction

Innovation

• Multi-token prediction enhances speech-to-unit translation
• MTP loss applied to hidden representations with CTC
• Earlier semantic enrichment improves translation quality