Benchmarking the Benchmarks: Evaluating Benchmarks for Conversational Agents

2026-08-06

AI Summary

Existing task-oriented dialogue evaluation benchmarks lack systematic quality assessment, potentially leading to unreliable results. This work proposes the first reference-free automatic evaluation framework that leverages large language models as judges to conduct fine-grained, interpretable quality diagnostics of both synthetic and human-constructed dialogue benchmarks across three dimensions: consistency, complexity, and strategic coverage. Through controlled perturbation experiments and multi-dimensional metrics, the framework effectively discriminates benchmark quality levels across multiple domains and diverse judge models. Its evaluations exhibit strong alignment with human annotations, demonstrating both validity and generalizability.

Abstract

Task-oriented conversational agents are evaluated using curated or automatically generated benchmarks, yet benchmark quality is rarely assessed. Poor benchmarks may contain inconsistent tasks, simplistic scenarios, or limited policy coverage, leading to unreliable evaluations. We introduce a reference-free framework that uses LLM judges to assess benchmark consistency, complexity, and policy coverage, while providing actionable diagnostics of weaknesses. We validate the framework by demonstrating agreement with independent human annotations and by evaluating benchmarks generated by LLMs of varying capabilities, as well as benchmarks subjected to controlled quality-degrading perturbations. Across domains and judge models, the proposed metrics consistently distinguish between benchmark quality levels. We further demonstrate the framework's applicability to manually curated benchmarks. Our framework offers a practical approach for evaluating synthetic and manually curated conversational-agent benchmarks.

Problem

• benchmark quality
• conversational agents
• evaluation reliability
• policy coverage
• task consistency

Innovation

• benchmark evaluation
• conversational agents
• LLM judges
• reference-free framework
• policy coverage