Evaluating, Synthesizing, and Enhancing for Customer Support Conversation

2025-08-06

AI Summary

Existing customer support dialogue datasets lack structured strategy guidance, while real-world service data is scarce and expensive to annotate—hindering the training of professional, empathy-oriented customer service agents. Method: We formalize the Customer Support Conversation (CSC) task and propose a five-phase, twelve-strategy framework grounded in COPC standards. Leveraging LLM-driven dialogue reconstruction and structured annotation, we release CSConv—a benchmark evaluation dataset. We further introduce RoleCS, a novel role-playing-based data synthesis method that generates high-quality, strategy-aligned training dialogues. Crucially, we incorporate the structured service strategy system into LLM supervised fine-tuning for the first time, enabling explicit modeling of strategy execution capability. Contribution/Results: Experimental results demonstrate that RoleCS-finetuned models achieve significant improvements in strategy adherence accuracy, issue resolution rate, and human-evaluated response quality and empathy.

Technology Category

• Natural Language Processing: Conversational AI/Dialog Systems
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Assisted, interactive, and conversational search

Abstract

Effective customer support requires not only accurate problem solving but also structured and empathetic communication aligned with professional standards. However, existing dialogue datasets often lack strategic guidance, and real-world service data is difficult to access and annotate. To address this, we introduce the task of Customer Support Conversation (CSC), aimed at training customer service agents to respond using well-defined support strategies. We propose a structured CSC framework grounded in COPC guidelines, defining five conversational stages and twelve strategies to guide high-quality interactions. Based on this, we construct CSConv, an evaluation dataset of 1,855 real-world customer-agent conversations rewritten using LLMs to reflect deliberate strategy use, and annotated accordingly. Additionally, we develop a role-playing approach that simulates strategy-rich conversations using LLM-powered roles aligned with the CSC framework, resulting in the training dataset RoleCS. Experiments show that fine-tuning strong LLMs on RoleCS significantly improves their ability to generate high-quality, strategy-aligned responses on CSConv. Human evaluations further confirm gains in problem resolution. All code and data will be made publicly available at https://github.com/aliyun/qwen-dianjin.

Problem

• Enhancing customer support dialogue quality with strategic guidance
• Addressing lack of structured datasets for service conversations
• Improving LLM responses using strategy-aligned training data

Innovation

• Structured CSC framework with COPC guidelines
• CSConv dataset with LLM-rewritten strategy annotations
• RoleCS training via LLM-powered role-playing