asr evaluation

Current automatic speech recognition (ASR) evaluation metrics, such as word error rate (WER), often fail to accurately reflect human perception of transcription quality and lack user-centered assessment criteria. To address this gap, this study presents the first systematic construction of HATS, a human preference dataset for French ASR, comprising subjective pairwise preference annotations from 143 participants. The authors conduct a comprehensive correlation analysis between human judgments and both lexical-level and embedding-level metrics—including BERTScore and semantic distance—to evaluate their alignment with human perception. The findings reveal the extent to which prevailing ASR metrics correspond to human assessments, offering empirical insights and an open benchmark to support the development of more perceptually aligned evaluation methodologies.

This work addresses the challenge of incomparable evaluations and irreproducible results in speech understanding models, which often arise from discrepancies in post-processing, data handling, and pipeline design during deployment-oriented model selection. To this end, the authors propose SURE, a unified experimental framework that enables fair evaluation across diverse paradigms—from conventional pipelines to speech large language models—under realistic acoustic and linguistic stressors. SURE achieves this through standardized prediction formats, consistent normalization strategies, and a unified scoring mechanism. Furthermore, it introduces an agent-assisted training conversion pipeline that automatically maps published code into versioned, executable training workflows. This study presents the first unified and reproducible approach for both evaluating and training speech understanding systems across modeling paradigms, substantially enhancing comparability and reproducibility in real-world deployment scenarios.

Current speech-language models (SLMs) lack systematic evaluation tools for non-semantic acoustic attributes—such as background noise, speaker identity, emotion, and room impulse response—and their interplay with textual content. To address this, we introduce SALMon, the first unified benchmarking suite explicitly designed to assess SLMs across diverse non-semantic acoustic dimensions. SALMon employs discriminative scoring instead of generative sampling, significantly improving evaluation efficiency and consistency. It integrates a high-quality, expert-annotated acoustic attribute dataset with multi-dimensional, controllable acoustic perturbation injection to jointly evaluate both acoustic attribute fidelity and text-acoustic alignment. We conduct comprehensive benchmarking across state-of-the-art SLMs, precisely characterizing their capabilities and limitations along each acoustic dimension. All code and data are publicly released to foster reproducible research and community advancement.

Conventional automatic speech recognition (ASR) evaluation using word error rate (WER) fails to capture the practical impact of ASR errors on downstream large language model (LLM)-driven tasks. Method: We propose a task-oriented ASR evaluation framework that (1) systematically classifies ASR error types and analyzes their contextual reparability within LLM prompts; (2) defines a multidimensional metric integrating semantic severity of errors, LLM-based correction success rate, and end-task completion accuracy; and (3) validates the framework empirically on representative speech-to-LLM pipelines—including voice command execution and meeting summary generation. Results: Our framework significantly outperforms WER in reflecting ASR effectiveness in real-world LLM applications. It provides an interpretable, quantifiable assessment grounded in downstream task performance, enabling principled, task-aware ASR model development and optimization.

Traditional automatic speech recognition evaluation metrics, such as word error rate (WER) and character error rate (CER), fail to capture human perception of errors and neglect linguistic and semantic influences. This work proposes a novel paradigm that embeds any perception-oriented evaluation metric into the minimum edit distance (minED) framework to produce an intuitively interpretable equivalent error rate. For the first time, this approach translates human perceptual modeling into a comprehensible error rate format, enabling quantification of error severity from the perspective of human understanding. The resulting metric not only aligns closely with human judgments but also effectively identifies recognition errors that critically impact semantic comprehension.

This study addresses the limitations of traditional automatic speech recognition (ASR) evaluation, which relies heavily on word error rate (WER) and fails to capture the grammatical and semantic characteristics of transcription errors. To overcome this, the authors propose two novel metrics: Part-of-Speech Error Rate (POSER) and Embedding Error Rate (EmbER), which quantify ASR output quality from the perspectives of grammatical correctness and semantic fidelity, respectively. By integrating language model rescoring, part-of-speech tagging, and semantic distance computation based on word embeddings, they construct a multidimensional qualitative evaluation framework. Experimental results demonstrate that these new metrics effectively reveal the contribution of language models to improving linguistic quality in transcriptions, thereby compensating for WER’s insufficiency in linguistic analysis.

Conventional wisdom holds that human auditory recognition performance represents the upper bound for automatic speech recognition (ASR) systems, yet this assumption lacks systematic validation in diverse speech contexts—such as those involving children, older adults, and regional accents. This study presents the first systematic comparison between native Dutch listeners and state-of-the-art ASR systems (e.g., Google Telephony) on authentic, diverse Dutch speech data, examining the effects of speaker age, regional accent, and utterance length. Results reveal that ASR performance is generally on par with human listeners and even surpasses them under certain conditions, with system accuracy highly sensitive to test set composition. These findings challenge the long-standing belief that humans consistently outperform machines in speech recognition and underscore the critical need to enhance ASR robustness to speaker age and accent variation.

Existing evaluation methods for audio captioning struggle to accurately assess the fidelity of multimodal semantics and acoustic attributes in structured audio descriptions. This work proposes the first multi-axis evaluation framework tailored for structured audio captioning, integrating large language model (LLM)-based semantic judgments with deterministic acoustic metrics across five orthogonal dimensions: label sets, descriptive content, logical reasoning, numerical measurements, and spectral contours. The framework incorporates a controlled perturbation protocol to validate its ability to distinguish between semantic preservation and acoustic distortion. Experiments on the AudioCards dataset demonstrate that the proposed approach effectively differentiates semantically consistent paraphrases from genuine errors, significantly outperforming existing methods in both reliability and sensitivity.

This study addresses the limitations of conventional character- or word-error-rate metrics in evaluating end-to-end automatic speech recognition (ASR) systems, which often fail to capture the full spectrum of transcription quality. Focusing on French, the work proposes a multidimensional evaluation framework that integrates both linguistic and acoustic perspectives to overcome the constraints of single-metric assessments. Through systematic comparisons of various subword tokenization strategies—such as Byte Pair Encoding (BPE)—and prominent self-supervised speech representation models within end-to-end ASR architectures, the research elucidates how these components influence transcription accuracy and fluency. The resulting framework not only offers a more comprehensive and application-oriented approach to ASR evaluation but also establishes an empirical foundation for optimizing downstream French ASR systems.