PiDA: Phonetically-Informed Data Augmentation for Robust Vietnamese Speech Translation

2026-06-11

AI Summary

This study addresses the degradation of neural machine translation performance in cascaded speech translation systems caused by phoneme confusion in Vietnamese automatic speech recognition (ASR), which leads to frequent substitution errors. The work presents the first systematic phoneme-level categorization of Vietnamese ASR errors, revealing that substitutions predominantly arise from phonetic similarity. To mitigate this issue, the authors propose PiDA, a phoneme-aware data augmentation method that leverages phoneme-based word embeddings to generate phonetically similar substitute words, thereby synthesizing ASR-like erroneous data to enhance model robustness. Experiments on the FLEURS dataset employ linear mixed-effects models to quantify error impact and fine-tune end-to-end translation models. Results demonstrate that PiDA improves BLEU scores by up to 2.04 on ASR-contaminated inputs while also yielding modest gains on clean text.

Abstract

Cascaded speech translation (ST) systems suffer from error propagation when Automatic Speech Recognition (ASR) outputs incorrect transcripts. We present the first systematic categorization of ASR errors for Vietnamese ST, classifying substitution errors by phonetic cause and quantifying their impact on downstream Neural Machine Translation (NMT) performance using Linear Mixed-Effects Modelling. We confirm that most ASR substitution errors arise from phonetic confusions rather than random noise, and that these phonetic errors significantly degrade ST quality. Motivated by this finding, we propose Phonetically-Informed Data Augmentation (PiDA), which generates ASR-like corruptions by substituting words with phonetically similar alternatives using phonetic word embeddings. Fine-tuning on a PiDA-augmented version of FLEURS Vietnamese-English improves translation of erroneous ASR outputs (up to +2.04 BLEU over standard fine-tuning) while also slightly improving clean-text performance.

Problem

• speech translation
• error propagation
• ASR errors
• phonetic confusion
• Neural Machine Translation

Innovation

• Phonetically-Informed Data Augmentation
• ASR error analysis
• Vietnamese speech translation
• phonetic word embeddings
• error propagation mitigation