DialSim: A Real-Time Simulator for Evaluating Long-Term Multi-Party Dialogue Understanding of Conversation Systems

2024-06-19

AI Summary

Existing dialogue system evaluation often overlooks real-world complexities—including real-time response requirements, multi-party interaction, and long-range contextual dependencies. Method: We propose DialSim, the first dialogue simulator supporting real-time constraints, dynamic multi-role modeling, and uncertainty awareness; it enables LLMs to improvise as TV drama characters in time-limited, multi-turn, multi-party dialogues grounded in long-range context while explicitly identifying knowledge boundaries. Concurrently, we introduce LongDialQA—a benchmark dataset enabling robust evaluation under dynamically randomized questioning. Our methodology integrates role-driven simulation design, context-aware response generation, real-time latency monitoring, and a structured evaluation framework. Contribution/Results: Comprehensive experiments expose critical bottlenecks in state-of-the-art models regarding temporal constraint adherence, role tracking fidelity, and knowledge boundary calibration. DialSim establishes a reproducible, quantitative benchmark for assessing long-horizon understanding and reasoning capabilities in interactive dialogue settings.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Conversational AI/Dialog Systems
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Recent advancements in Large Language Models (LLMs) have significantly enhanced the capabilities of conversation systems, making them applicable to various fields (e.g., education). Despite their progress, the evaluation of the systems often overlooks the complexities of real-world conversations, such as real-time interactions, multi-party dialogues, and extended contextual dependencies. To bridge this gap, we introduce DialSim, a real-time dialogue simulator. In this simulator, a conversation system is assigned the role of a character from popular TV shows, requiring it to respond to spontaneous questions using past dialogue information and to distinguish between known and unknown information. Key features of DialSim include assessing the system's ability to respond within a reasonable time limit, handling long-term multi-party dialogues, and evaluating performance under randomized questioning with LongDialQA, a novel, high-quality question-answering dataset. Our experiments using DialSim reveal the strengths and weaknesses of the latest conversation systems, offering valuable insights for future advancements in conversational AI. DialSim is available at https://dialsim.github.io/.

Problem

• evaluating real-time multi-party dialogue understanding
• assessing long-term contextual dependencies in conversations
• testing system performance with randomized questioning

Innovation

• Real-time dialogue simulator
• Multi-party dialogue handling
• Long-term context evaluation