🤖 AI Summary
This work addresses the limitations of conventional automatic speech recognition (ASR) evaluation metrics—such as word error rate (WER)—which exhibit poor performance in low-resource languages and hinder fair cross-lingual comparisons. To overcome these challenges, the authors propose an enhanced WER evaluation framework that integrates language-specific text normalization, compound word detection, and a fine-grained token-level Levenshtein distance-based alignment mechanism. Notably, this is the first open-source implementation to incorporate an alignment representation capable of embedding metadata. Evaluated across 52 languages, the proposed method achieves absolute WER reductions of up to 25%, substantially improving the robustness and fairness of cross-lingual ASR evaluation.
📝 Abstract
Advances in deep learning and end-to-end Automatic Speech Recognition (ASR) have enabled robust multilingual models, but evaluation metrics remain limited in assessing accuracy. Efforts to improve or replace the common metric Word Error Rate (WER) often focus on English, leaving evaluations for low-resource languages under-explored and hindering fair cross-lingual comparisons. We present OpenWER, an open-source implementation that improves WER robustness through language-specific normalisation and compound word detection. A token-based Levenshtein alignment preserves complementary metrics and allows metadata embedding for granular accuracy scores. Our analysis of 52 languages shows absolute WER reductions of up to 25% compared to common libraries. OpenWER contributes to fairness in ASR research by increasing the reliability of WER across diverse languages and enabling more comprehensive accuracy evaluations.