Technology Category
Natural Language Processing: Conversational AI/Dialog SystemsMachine Learning: Large Multimodal Models (LMMs)Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
TELEVAL: A Dynamic Benchmark Designed for Spoken Language Models in Chinese Interactive Scenarios
🤖 AI Summary
Existing spoken language model (SLM) benchmarks primarily evaluate performance on complex, task-oriented scenarios, neglecting semantic understanding and responsive capabilities essential for natural, user-initiated dialogue.
Method: We propose TELEVAL, a dynamic, Chinese-specific evaluation benchmark designed for interactive settings. It assesses SLMs across three dimensions—explicit semantics, paralinguistic and implicit semantics, and system-level capabilities—in instruction-free, multi-turn, multimodal (text + audio) dialogues. TELEVAL employs realistic dialogue formats and dynamic interaction protocols, enabling the first fine-grained evaluation of implicit intent recognition and context-adaptive response generation.
Results: Experiments reveal substantial deficiencies in current SLMs’ natural conversational competence. TELEVAL effectively discriminates models based on authentic interactive capability, thereby bridging a critical gap in user-experience-aligned evaluation.
Technology Category
Application Category
📝 Abstract
Spoken language models (SLMs) have seen rapid progress in recent years, along with the development of numerous benchmarks for evaluating their performance. However, most existing benchmarks primarily focus on evaluating whether SLMs can perform complex tasks comparable to those tackled by large language models (LLMs), often failing to align with how users naturally interact in real-world conversational scenarios. In this paper, we propose TELEVAL, a dynamic benchmark specifically designed to evaluate SLMs' effectiveness as conversational agents in realistic Chinese interactive settings. TELEVAL defines three evaluation dimensions: Explicit Semantics, Paralinguistic and Implicit Semantics, and System Abilities. It adopts a dialogue format consistent with real-world usage and evaluates text and audio outputs separately. TELEVAL particularly focuses on the model's ability to extract implicit cues from user speech and respond appropriately without additional instructions. Our experiments demonstrate that despite recent progress, existing SLMs still have considerable room for improvement in natural conversational tasks. We hope that TELEVAL can serve as a user-centered evaluation framework that directly reflects the user experience and contributes to the development of more capable dialogue-oriented SLMs.
Problem
Research questions and friction points this paper is trying to address.
Evaluating SLMs in realistic Chinese conversational scenarios
Assessing SLMs' ability to extract implicit speech cues
Aligning SLM benchmarks with natural user interaction patterns
Innovation
Methods, ideas, or system contributions that make the work stand out.
Dynamic benchmark for Chinese SLM evaluation
Three evaluation dimensions: explicit, implicit, system
Separate text and audio output assessment
🔎 Similar Papers
No similar papers found.
