Distributional Metrics for Evaluating Spoken Conversational Systems

2026-09-21

AI Summary

This study addresses the challenge of evaluating spoken dialogue systems in the absence of objective criteria by proposing the Conversation Similarity (CDS) metric. Taking natural human conversation as a reference distribution, CDS extracts eight-dimensional behavioral features—including speech rate, rhythm, and interaction dynamics—and integrates them with semantic embedding analysis to quantify discrepancies between system-generated and human dialogues via statistical distribution comparison techniques. By adopting a distribution-comparison perspective, CDS provides an interpretable, multidimensional evaluation framework that effectively reproduces the preference rankings of most listeners. The findings demonstrate that this metric serves as a valuable complement to conventional evaluation methods, offering a new paradigm for the objective assessment of spoken dialogue systems.

Abstract

Evaluating conversational systems is a difficult and unresolved problem. We introduce the Conversational Distribution Score (CDS), which compares distributions of conversational behaviour using human conversations as a reference. CDS describes speech rate, syllabic rhythm, and turn interaction through eight interpretable features plus a separate two-feature semantic baseline. We compare conversations with two reference scales: one based on conversational success within human dialogue and another contrasting human and synthetic dialogue. Using listener judgments from out-of-domain goal--oriented dialogues, we examine system ranking, preferences between conversations, and ranking stability. Composite CDS recovers five of six listener system comparisons while individual features show strong correlation with listener preferences between conversations. We examine how many minutes and conversations are required before rankings stabilize. These findings support distributional comparisons as a complement to specific interactional metrics to evaluate conversations and conversational models while showing their interpretable value.

Problem

• spoken conversational systems
• evaluation
• distributional metrics
• conversational behavior

Innovation

• Conversational Distribution Score
• Distributional Metrics
• Spoken Conversational Systems
• Interpretable Features
• Dialogue Evaluation