Rethinking Human-Aligned Evaluation: An Analysis of Semantic Metrics Beyond WER

2026-09-18

AI Summary

研究通过引入HATS-en数据集，对比多种语义度量方法，发现WER与人类判断最不一致，建议使用CER和SemDist作为ASR评估指标。

Abstract

Word Error Rate (WER), the most commonly used metric for Automatic Speech Recognition (ASR), treats every lexical deviation from the reference as equally costly, regardless of whether it changes meaning. This raises the question: does WER actually track how humans judge ASR transcript quality? We introduce HATS-en, an English dataset for human-centered ASR evaluation. Using this dataset, we benchmark lexical metrics against several configurations of BERTScore and SemDist, varying the language model, layer, and pooling strategy. We find that WER agrees least with human judgment among all metrics tested, that the best-performing SemDist configurations achieve the highest overall agreement, ahead of CER and BERTScore, and that no single model is best across settings. CER, despite its simplicity and low cost, remains remarkably close to these best configurations. In line with prior recommendations, our results support shifting ASR evaluation toward CER both for English and for morphosyllabic writing systems as it is a more interpretable and low-cost metric for what evaluation should actually capture, and using SemDist as a complementary evaluation.

Problem

• Word Error Rate
• Automatic Speech Recognition
• Human Judgment
• Semantic Metrics

Innovation

• HATS-en
• SemDist
• CER