speech evaluation metrics

Subjective listening tests remain the bottleneck for evaluating speech quality of neural audio codecs at low bitrates. Method: We systematically benchmark mainstream objective metrics—including PESQ, STOI, and DNSMOS—against human perception using standardized MUSHRA subjective test results, quantifying their correlation with mean opinion scores via Pearson’s correlation coefficient. Contribution/Results: Traditional metrics (e.g., PESQ) exhibit markedly degraded performance under neural codec distortions, whereas DNSMOS and novel time-frequency domain metrics achieve superior correlation (r > 0.85). We are the first to characterize differential sensitivity of objective metrics to neural-specific artifacts—such as spectral smearing and temporal aliasing—and to propose an empirically grounded, optimized metric combination with clearly defined applicability boundaries for neural audio codecs. This work provides evidence-based guidelines for automated, reproducible speech quality assessment in neural codec development and evaluation.

Subjective Mean Opinion Score (MOS) ratings hinder scalable, generalizable speech quality assessment due to high annotation cost and poor cross-dataset transferability. Method: We propose a reference-free, self-supervised paradigm: (i) synthesizing paired speech samples with diverse acoustic degradations (e.g., noise, reverberation); (ii) extracting representations using WavLM; (iii) leveraging cosine similarity between clean-degraded pairs as self-supervised signal; and (iv) training a Transformer-based regressor to predict degradation severity. Contribution/Results: This work pioneers the integration of speech foundation models with self-supervised contrastive learning—eliminating reliance on MOS entirely. Experiments demonstrate strong correlation with MOS (ρ > 0.92) on unseen benchmarks (NISQA, VOiCES), high consistency with ASR performance and physical acoustic parameters (e.g., microphone distance), and robust cross-corpus generalization.

Current evaluation of speech reconstruction relies heavily on Mean Opinion Score (MOS) ratings, which struggle to reliably capture the trade-off between naturalness and speaker similarity for highly unintelligible speech. To address this limitation, this work proposes the first hybrid subjective–objective evaluation framework. Subjectively, it introduces a contextualized Best–Worst Scaling (BWS) paradigm to precisely assess intelligibility and speaker identity perception. Objectively, it designs a novel dual-reference distribution-based metric that effectively quantifies the relationship between these two dimensions. Experiments across 17 zero-shot text-to-speech systems and 193 speakers demonstrate that the proposed framework achieves high reliability and strong alignment with downstream tasks, significantly outperforming existing evaluation methods.

To address the lack of unified, standardized evaluation tools for speech, audio, and music signals, this paper introduces the first cross-task, cross-modal, and configurable lightweight evaluation toolkit. The toolkit integrates 65 metrics and 729 configurable variants, supporting multi-source reference evaluation—including waveforms, text transcriptions, and semantic descriptions—across five downstream tasks: audio coding, speech synthesis, speech enhancement, singing voice synthesis, and music generation. Leveraging a Pythonic API, modular metric encapsulation, dependency isolation, and multimodal fusion evaluation techniques, it enables out-of-the-box, end-to-end assessment of both perceptual quality and semantic consistency. Extensive validation on multiple benchmarks confirms its metric diversity and configuration flexibility. The toolkit is open-sourced and has been widely adopted by the research community.

This paper addresses long-overlooked bottlenecks in speech enhancement (SE): (1) bandwidth mismatch and implicit label noise in training corpora; (2) insufficient robustness under extreme conditions (e.g., speaker overlap, high noise/reverberation) and lack of quantifiable metrics for hard samples; and (3) poor correlation between single objective metrics and subjective perceptual quality. We propose a data quality diagnostic framework with bandwidth consistency verification, revealing—for the first time—systematic effective bandwidth deviations and >15% label noise across mainstream SE corpora. Furthermore, we introduce a difficulty-aware, multi-metric fusion evaluation framework that integrates objective measures with MOS-mapped weighted aggregation. Experiments demonstrate a 32% improvement in Pearson correlation (r) between automatic assessment and human judgments, significantly enhancing the reliability and interpretability of SE system development.

Traditional speech quality assessment methods (e.g., PESQ, POLQA) rely on controlled conditions and costly subjective listening tests, while data-driven models (e.g., NISQA) enable MOS and perceptual dimension regression but lack natural language interaction and interpretability. This paper proposes the first large language model (LLM)-based speech quality assessment framework, integrating an audio encoder with an LLM to support reference-free and reference-based multi-dimensional natural language question answering. The framework jointly predicts subjective MOS scores, simulates listener perception differences, and generates textual explanations. Evaluated on the NISQA dataset, our method constructs QA pairs via templating and jointly optimizes regression and ranking losses. In a dual-encoder setup, it achieves a MOS prediction MAE of 0.41 and a Pearson correlation coefficient of 0.86, with strong performance across all perceptual dimensions—significantly reducing reliance on crowdsourced subjective testing.

This work addresses the limited correlation between traditional signal-to-noise ratio (SNR) and human auditory perception in audio generation tasks, which hinders effective quality assessment. To overcome this limitation, the study introduces phase distance into the SNR framework for the first time, reformulating the SNR metric based on signal processing theory to propose a novel evaluation measure, GOMPSNR. Furthermore, two loss functions are designed: magnitude-guided phase optimization and joint magnitude-phase optimization. Experimental results demonstrate that GOMPSNR more accurately quantifies audio distortion, and the proposed loss functions significantly improve the generation quality of neural vocoders, thereby enhancing the alignment between objective metrics and subjective perceptual judgments.

This study addresses the limited generalization of existing objective speech quality assessment models on unseen data, which hinders their reliable deployment. To this end, the authors propose a Dataset-Split Concealment (DSC) evaluation paradigm, wherein a subset of datasets is withheld during training to simulate real-world unseen conditions and systematically quantify model generalization. Additionally, they introduce a lightweight Aligner module—adding only 1,000 parameters—to effectively mitigate corpus-specific biases. Evaluated across nine training sets and nine unseen test sets, the DSC framework demonstrates consistent improvements when integrated with established models such as MOSNet, NISQA, and Wav2Vec2.0. Notably, the Aligner significantly enhances the accuracy of speech quality estimation on unseen data, particularly for large-scale models like Wav2Vec2.0 (94 million parameters).

This study addresses the limitations of current speech quality assessment models in capturing perceptual differences arising from prosodic errors and variations in speaker characteristics such as fundamental frequency and speaking rate. By systematically introducing acoustic degradations, prosodic distortions, and speaker attribute modifications through controlled perturbations, the authors compare human subjective ratings with predictions from state-of-the-art mean opinion score (MOS) models. Their analysis reveals that existing models are generally insensitive to prosodic errors, exhibit artificial biases toward fundamental frequency, and fail to respond adequately to changes in speaking rate and its variability. These findings challenge the prevailing paradigm of relying solely on scalar MOS for speech quality evaluation, highlighting the unidimensionality of current approaches and providing empirical grounding for the development of more perceptually aligned, multidimensional speech quality assessment frameworks.

The proliferation of deepfake speech and the absence of standardized evaluation protocols for detection methods hinder progress in audio security. Method: This work establishes the first large-scale, cross-domain benchmark for fake speech detection, systematically evaluating eight state-of-the-art detectors on speech synthesized by 20 mainstream text-to-speech systems. It introduces a reproducible adversarial testing framework to assess real-world robustness. Contribution/Results: We identify severe cross-domain performance degradation and critical security vulnerabilities in current detectors. To address the lack of standardized evaluation, we propose a comprehensive, multi-dimensional metric suite—incorporating generalization, domain robustness, and adversarial resilience. Our open-source, extensible benchmark enables rigorous, comparable assessment and facilitates the shift from isolated “point defenses” toward holistic system-level robustness in trustworthy AI-based speech security.