Towards Orthographically-Informed Evaluation of Speech Recognition Systems for Indian Languages

2026-03-01

AI Summary

Traditional word error rate (WER) significantly overestimates errors in evaluating speech recognition systems for Indian languages due to spelling variants, flexible suffix segmentation, and non-standard orthography, thereby failing to reflect actual user experience. This work proposes the first orthography-aware WER metric, termed OIWER, which integrates orthographic variation into ASR evaluation by leveraging large language models (LLMs) to generate acceptable spelling variants and construct a more robust evaluation benchmark. Experimental results demonstrate that OIWER reduces error rates by an average of 6.3 percentage points, narrows performance gaps between models—such as decreasing the Gemini–Canary gap from 18.1 to 11.5—and achieves 4.9 points higher agreement with human judgments compared to WER-SN.

Technology Category

• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Evaluation and Analysis

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success

Abstract

Evaluating ASR systems for Indian languages is challenging due to spelling variations, suffix splitting flexibility, and non-standard spellings in code-mixed words. Traditional Word Error Rate (WER) often presents a bleaker picture of system performance than what human users perceive. Better aligning evaluation with real-world performance requires capturing permissible orthographic variations, which is extremely challenging for under-resourced Indian languages. Leveraging recent advances in LLMs, we propose a framework for creating benchmarks that capture permissible variations. Through extensive experiments, we demonstrate that OIWER, by accounting for orthographic variations, reduces pessimistic error rates (an average improvement of 6.3 points), narrows inflated model gaps (e.g., Gemini-Canary performance difference drops from 18.1 to 11.5 points), and aligns more closely with human perception than prior methods like WER-SN by 4.9 points.

Problem

• speech recognition evaluation
• orthographic variation
• Indian languages
• Word Error Rate
• code-mixed text

Innovation

• Orthographically-Informed Evaluation
• Speech Recognition
• Indian Languages
• LLM-based Benchmarking
• OIWER