Technology Category
Natural Language Processing: SpeechMachine Learning: Evaluation and AnalysisCognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning
SpeechIQ: Speech Intelligence Quotient Across Cognitive Levels in Voice Understanding Large Language Models
🤖 AI Summary
Current speech evaluation overly relies on Word Error Rate (WER), failing to reflect models’ true speech understanding capabilities in memory, comprehension, and application. To address this, we propose Speech-based Intelligence Quotient (SIQ)—the first three-tier speech understanding evaluation framework inspired by Bloom’s Taxonomy, uniformly applicable to both cascaded and end-to-end models. SIQ integrates automatic speech recognition accuracy, semantic similarity, and downstream question-answering performance to enable quantifiable, cognition-grounded assessment. Experiments demonstrate that SIQ effectively uncovers annotation errors and model hallucinations in mainstream benchmarks, reveals latent deficiencies in multimodal training, and enables fair cross-architecture model comparison. By grounding evaluation in cognitive principles, SIQ significantly enhances the conceptual validity and practical utility of speech intelligence assessment.
Technology Category
Application Category
📝 Abstract
We introduce Speech-based Intelligence Quotient (SIQ) as a new form of human cognition-inspired evaluation pipeline for voice understanding large language models, LLM Voice, designed to assess their voice understanding ability. Moving beyond popular voice understanding metrics such as word error rate (WER), SIQ examines LLM Voice across three cognitive levels motivated by Bloom's Taxonomy: (1) Remembering (i.e., WER for verbatim accuracy); (2) Understanding (i.e., similarity of LLM's interpretations); and (3) Application (i.e., QA accuracy for simulating downstream tasks). We demonstrate that SIQ not only quantifies voice understanding abilities but also provides unified comparisons between cascaded methods (e.g., ASR LLM) and end-to-end models, identifies annotation errors in existing benchmarks, and detects hallucinations in LLM Voice. Our framework represents a first-of-its-kind intelligence examination that bridges cognitive principles with voice-oriented benchmarks, while exposing overlooked challenges in multi-modal training.
Problem
Research questions and friction points this paper is trying to address.
Assessing voice understanding in LLMs using cognitive levels
Moving beyond WER to evaluate LLM Voice performance
Unifying comparisons between cascaded and end-to-end models
Innovation
Methods, ideas, or system contributions that make the work stand out.
Introduces Speech-based Intelligence Quotient (SIQ) evaluation
Assesses LLM Voice across three cognitive levels
Unifies comparisons between cascaded and end-to-end models
🔎 Similar Papers
No similar papers found.
