"Stupid robot, I want to speak to a human!" User Frustration Detection in Task-Oriented Dialog Systems

2024-11-26

arXiv.org

AI Summary

Detecting user frustration in real-world task-oriented dialogue systems remains challenging due to the implicit, context-dependent, and linguistically diverse nature of frustration expressions. Method: This work presents the first systematic evaluation of large language models (LLMs) with in-context learning for industrial-scale frustration detection. We benchmark conventional approaches—including keyword-based rules, VADER/TextBlob sentiment analyzers, and dialogue breakdown detectors—against LLM-based zero-shot and few-shot classification on internal production data. Contribution/Results: Our LLM-driven approach achieves a 16% F1-score improvement over baselines on an internal benchmark, demonstrating practical deployability. Furthermore, we introduce a reusable analytical framework for frustration pattern identification and a practitioner-oriented implementation guide, establishing a novel paradigm for enhancing user experience in task-oriented dialogue systems.

Technology Category

Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)Machine Learning: Large Multimodal Models (LMMs)Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendationEconomics, Online Markets and Human Computation: Cost models of using LLMs in production systemsSearch and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Detecting user frustration in modern-day task-oriented dialog (TOD) systems is imperative for maintaining overall user satisfaction, engagement, and retention. However, most recent research is focused on sentiment and emotion detection in academic settings, thus failing to fully encapsulate implications of real-world user data. To mitigate this gap, in this work, we focus on user frustration in a deployed TOD system, assessing the feasibility of out-of-the-box solutions for user frustration detection. Specifically, we compare the performance of our deployed keyword-based approach, open-source approaches to sentiment analysis, dialog breakdown detection methods, and emerging in-context learning LLM-based detection. Our analysis highlights the limitations of open-source methods for real-world frustration detection, while demonstrating the superior performance of the LLM-based approach, achieving a 16% relative improvement in F1 score on an internal benchmark. Finally, we analyze advantages and limitations of our methods and provide an insight into user frustration detection task for industry practitioners.

Problem

• Task-oriented Dialog Systems
• User Frustration Detection
• Service Quality Improvement

Innovation

• Emotion Recognition
• User Dissatisfaction
• AI-enhanced Dialogue Systems