Rethinking MUSHRA: Addressing Modern Challenges in Text-to-Speech Evaluation

2024-11-19

arXiv.org

AI Summary

Existing TTS human evaluation methods suffer from three key limitations: MOS lacks discriminative power; CMOS is inefficient for large-scale assessment; and MUSHRA systematically penalizes superhuman-quality synthesis due to its mandatory anchoring to human reference recordings, while also being vulnerable to rater variability, auditory fatigue, and reference bias. To address these issues, we conduct a large-scale subjective study with 492 Hindi and Tamil listeners and propose two innovations: (1) a reference-decoupled MUSHRA variant that eliminates unjustified penalties against synthesis exceeding human quality; and (2) a structured rating guideline that substantially reduces inter-rater score variance. Concurrently, we introduce MANGO—the first high-quality, multilingual TTS subjective evaluation benchmark—comprising 246,000 human ratings on Indian language speech. MANGO enables fairer, finer-grained, and more reliable TTS assessment and facilitates robust automatic metric development.

Technology Category

• Natural Language Processing: Speech
• Machine Learning: Evaluation and Analysis
• Humans and AI: Crowd Sourcing and Human Computation

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Despite rapid advancements in TTS models, a consistent and robust human evaluation framework is still lacking. For example, MOS tests fail to differentiate between similar models, and CMOS's pairwise comparisons are time-intensive. The MUSHRA test is a promising alternative for evaluating multiple TTS systems simultaneously, but in this work we show that its reliance on matching human reference speech unduly penalises the scores of modern TTS systems that can exceed human speech quality. More specifically, we conduct a comprehensive assessment of the MUSHRA test, focusing on its sensitivity to factors such as rater variability, listener fatigue, and reference bias. Based on our extensive evaluation involving 492 human listeners across Hindi and Tamil we identify two primary shortcomings: (i) reference-matching bias, where raters are unduly influenced by the human reference, and (ii) judgement ambiguity, arising from a lack of clear fine-grained guidelines. To address these issues, we propose two refined variants of the MUSHRA test. The first variant enables fairer ratings for synthesized samples that surpass human reference quality. The second variant reduces ambiguity, as indicated by the relatively lower variance across raters. By combining these approaches, we achieve both more reliable and more fine-grained assessments. We also release MANGO, a massive dataset of 246,000 human ratings, the first-of-its-kind collection for Indian languages, aiding in analyzing human preferences and developing automatic metrics for evaluating TTS systems.

Problem

• Lack of robust human evaluation framework for TTS models
• MUSHRA test penalizes modern TTS systems exceeding human quality
• Reference bias and judgment ambiguity in TTS evaluation

Innovation

• Refined MUSHRA test variants for fairer ratings
• Addressed reference bias and judgement ambiguity issues
• Released MANGO dataset for TTS evaluation metrics