Limitations of Current Evaluation Practices for Conversational Recommender Systems and the Potential of User Simulation

2025-10-07

AI Summary

Current conversational recommendation systems (CRS) face two critical evaluation bottlenecks: static test sets fail to capture real-world interaction dynamics, and mainstream automatic metrics exhibit poor correlation with actual user satisfaction. To address these issues, we propose a user-simulation-based dynamic evaluation paradigm, introducing a novel reward-cost-centered evaluation framework. Our approach integrates multi-strategy user simulation with empirical analysis of real user behavior to generate high-fidelity interactive data—replacing conventional static test sets. Experiments demonstrate a substantial improvement in ranking consistency with human evaluation (+23.6% Spearman correlation), and—crucially—provide the first empirical identification of systematic biases inherent in existing automatic metrics. This work establishes a reproducible, user-centric methodology for CRS evaluation, shifting the focus from model-centric to experience-centric assessment.

Technology Category

• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Machine Learning: Learning Preferences or Rankings
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Economics and fairness of platforms and recommendation systems

Abstract

Research and development on conversational recommender systems (CRSs) critically depends on sound and reliable evaluation methodologies. However, the interactive nature of these systems poses significant challenges for automatic evaluation. This paper critically examines current evaluation practices and identifies two key limitations: the over-reliance on static test collections and the inadequacy of existing evaluation metrics. To substantiate this critique, we analyze real user interactions with nine existing CRSs and demonstrate a striking disconnect between self-reported user satisfaction and performance scores reported in prior literature. To address these limitations, this work explores the potential of user simulation to generate dynamic interaction data, offering a departure from static datasets. Furthermore, we propose novel evaluation metrics, based on a general reward/cost framework, designed to better align with real user satisfaction. Our analysis of different simulation approaches provides valuable insights into their effectiveness and reveals promising initial results, showing improved correlation with system rankings compared to human evaluation. While these findings indicate a significant step forward in CRS evaluation, we also identify areas for future research and refinement in both simulation techniques and evaluation metrics.

Problem

• Current CRS evaluation relies excessively on static datasets
• Existing metrics fail to align with real user satisfaction
• User simulation addresses limitations by generating dynamic interactions

Innovation

• User simulation generates dynamic interaction data
• Novel metrics align with real user satisfaction
• General reward-cost framework improves evaluation correlation