Dissecting Sensitivity to Training Language in Self-Supervised Speech Learning Using Neural Audio Codec Tokens

2026-07-28

AI Summary

This study investigates the language sensitivity of neural audio codecs (NACs) and self-supervised learning (SSL) speech models in multilingual settings, addressing whether separate models must be trained for each language. By fixing the pretraining language of either the NAC or SSL model and systematically evaluating downstream task performance, the work reveals—for the first time—that the NAC’s training language has negligible impact on performance, whereas alignment between the SSL pretraining language and the target language is critical. These findings demonstrate that a single NAC can be effectively reused across languages, substantially reducing the training cost of multilingual SSL systems without compromising performance. This insight establishes a new paradigm for efficiently building multilingual speech models.

Abstract

Neural audio codecs (NACs) have become popular for obtaining speech representations as discrete tokens. Beyond compression, discrete tokens can be used to train self-supervised learning (SSL) models. Such models, referred to as codec-based SSL models, reduce data storage and computational cost, enabling scalable SSL pre-training. However, their language sensitivity remains unclear. When the language changes, codec-based SSL models may require retraining, which undermines their efficiency. In this paper, we present a systematic analysis of language sensitivity by varying either the NAC training language or the SSL pre-training language while keeping the other fixed. Experimental results show that downstream performance is insensitive to the NAC training language but strongly dependent on the SSL pre-training language. These findings suggest that a single NAC can be reused across languages, while aligning the SSL pre-training language with the target language is crucial.

Problem

• language sensitivity
• self-supervised speech learning
• neural audio codec
• discrete tokens
• SSL pre-training

Innovation

• neural audio codec
• self-supervised learning
• language sensitivity
• speech representation
• discrete tokens