Latent Softmax for Data-Efficient Phoneme-Based Multilingual ASR Across Tonal and Non-Tonal Languages

2026-08-02

AI Summary

This work addresses the challenge in multilingual speech recognition that differing phoneme annotation granularities between tonal and non-tonal languages hinder cross-lingual acoustic sharing. To reconcile this, the authors propose a Latent Softmax output layer within a CTC framework, which models tone-marked vowels as subclasses and their underlying base vowels as superclasses. When tone is unobserved, it is treated as a latent variable and marginalized out, enabling a unified hierarchical phoneme representation. This approach introduces hierarchical modeling to multilingual ASR for the first time, balancing the discriminative needs of tonal languages with the data efficiency required for non-tonal ones, thereby significantly enhancing cross-lingual transfer. Experiments show relative S2P error rate reductions of 8.4%, 17.5%, and 12.6% on AISHELL-1 and LibriSpeech, with further mixed error rate improvements of 2.6%–9.5% in code-switching scenarios.

Abstract

Phoneme-based multilingual automatic speech recognition (ASR) can share acoustic evidence across languages more directly than language-specific subword modeling. When tonal and non-tonal languages are jointly trained, however, their supervision granularity does not match: tonal languages annotate tone-marked vowels, whereas non-tonal languages typically provide only base-vowel labels. A standard softmax either treats the two as unrelated classes, weakening cross-lingual sharing, or collapses tones, losing distinctions required by tonal languages. We propose Latent Softmax, a connectionist temporal classification (CTC)-compatible output layer that models tone-marked vowels as subclasses and base vowels as major classes, while consonants and the CTC blank remain singleton labels. When only a base-vowel major-class label is observed, the tone-marked vowel subclass is treated as latent and marginalized out. Multilingual experiments on AISHELL-1 Mandarin and LibriSpeech English show that Latent Softmax reduces speech-to-phoneme (S2P) phoneme error rates over a standard softmax multilingual baseline by 8.4% on AISHELL-1, 17.5% on LibriSpeech test-clean, and 12.6% on test-other. The improved speech-to-phoneme encoders also yield consistent word error rate gains for both large-language-model phoneme-to-grapheme conversion and projector-based interfaces. After code-switching adaptation, Latent Softmax further reduces projector-based mixed error rate by 2.6% on ASRU2019 and 9.5% on CS-Dialogue datasets.

Problem

• multilingual ASR
• tonal languages
• phoneme-based modeling
• label granularity mismatch
• cross-lingual sharing

Innovation

• Latent Softmax
• phoneme-based multilingual ASR
• tonal languages
• connectionist temporal classification
• cross-lingual sharing