Turning Conversations into Workflows: A Framework to Extract and Evaluate Dialog Workflows for Service AI Agents

2025-02-24

AI Summary

Service workflows in customer service dialogues are often missing and unstructured, leading to inconsistent AI responses. Method: This paper proposes the first integrated framework for automatic dialogue workflow extraction and simulation-based evaluation. It combines retrieval-augmented generation (RAG) with a novel question-answering–based chain-of-thought (QA-CoT) prompting technique to improve structured workflow generation accuracy. Additionally, it introduces a scalable two-agent (agent + customer) simulation evaluation mechanism for automated, large-scale workflow assessment. Contribution/Results: We design a macro-accuracy metric that achieves high agreement with human evaluation (Spearman ρ > 0.92). On the ABCD and SynthABCD datasets, our method improves average macro-accuracy by 12.16%, significantly mitigating response inconsistency caused by workflow omission in service AI systems.

Technology Category

• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Natural Language Processing: Conversational AI/Dialog Systems
• Multiagent Systems: Agent-Based Simulation and Emergent Behavior

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Economics, Online Markets and Human Computation: Architectures and workflows that use LLMs for crowd work

Abstract

Automated service agents require well-structured workflows to provide consistent and accurate responses to customer queries. However, these workflows are often undocumented, and their automatic extraction from conversations remains unexplored. In this work, we present a novel framework for extracting and evaluating dialog workflows from historical interactions. Our extraction process consists of two key stages: (1) a retrieval step to select relevant conversations based on key procedural elements, and (2) a structured workflow generation process using a question-answer-based chain-of-thought (QA-CoT) prompting. To comprehensively assess the quality of extracted workflows, we introduce an automated agent and customer bots simulation framework that measures their effectiveness in resolving customer issues. Extensive experiments on the ABCD and SynthABCD datasets demonstrate that our QA-CoT technique improves workflow extraction by 12.16% in average macro accuracy over the baseline. Moreover, our evaluation method closely aligns with human assessments, providing a reliable and scalable framework for future research.

Problem

• Extract dialog workflows from conversations
• Evaluate workflow quality automatically
• Improve accuracy using QA-CoT technique

Innovation

• Retrieval of relevant conversations
• QA-CoT for structured workflow
• Automated simulation for evaluation