dialogue simulation

This study addresses the weak interactive capability of large language models (LLMs) in multi-turn dialogue. Methodologically, it introduces a unified “capability–evaluation–enhancement–evolution” analytical framework—the first to jointly characterize context retention, dynamic response generation, and interactive modeling. The approach integrates dialogue state tracking, long-context modeling, trajectory-driven evaluation metrics, retrieval-augmented generation, and memory mechanisms, yielding a scalable multi-turn evaluation protocol and a collaborative interactive evolution pathway. The work provides the first systematic survey of LLM interactivity, precisely identifying core bottlenecks—including state drift, long-range forgetting, and evaluation misalignment—and offers both theoretical foundations and practical guidelines for applications such as conversational search, intelligent consulting, and interactive pedagogy.

This work addresses the evaluation and enhancement of large language models (LLMs) in multi-turn interactive settings across realistic domains—including mathematics, programming, healthcare, education, and adversarial jailbreaking—where key challenges involve long-horizon contextual consistency, response robustness, and fairness. We propose the first multidimensional benchmark taxonomy specifically designed for multi-turn dialogue, unifying three technical paradigms: intrinsic model capabilities, external augmentations (e.g., retrieval, memory, knowledge graphs), and agent-level coordination. We construct a structured evaluation resource repository and open-source an extensible challenge framework alongside practical guidelines (Awesome-Multi-Turn-LLMs). Our contributions provide a standardized benchmark, a systematic methodology, and reproducible baselines—advancing rigorous, comparable, and scalable research on multi-turn LLM interaction.

Existing large language models inadequately simulate authentic multi-turn human user behaviors—such as informal phrasing, personalized stylistic expression, and real-time self-correction—leading to biased and unrealistic evaluations of assistant models. Method: We propose User Language Models (User LMs), trained via human-centric post-training on multi-turn dialogue data—not by naïvely inverting assistant models—to explicitly capture realistic user interaction patterns. Contribution/Results: User LMs significantly improve behavioral fidelity and evaluation robustness, validated through both automated metrics and human assessments. Experiments reveal that when evaluated using User LMs, GPT-4o’s accuracy drops from 74.6% to 57.4% on programming and mathematical reasoning tasks, uncovering previously masked interaction weaknesses. This work establishes the first systematic, trustworthy user-side simulation paradigm for dialogue evaluation, introducing a new benchmark for assessing conversational capabilities of large models.

Public online security awareness remains low, and users struggle to recognize deceptive conversational patterns in social engineering scams. Method: This study proposes an interactive, large language model (LLM)-driven security education framework. It employs a dual-LLM adversarial simulation architecture to autonomously generate high-fidelity, diverse scam dialogues; integrates principles from learning science—including just-in-time feedback and guided reflection—to help users identify linguistic cues, assess risks, and practice defensive responses. Contribution/Results: Empirical evaluation demonstrates significant improvements in both scam detection accuracy and willingness to enact protective behaviors. This work constitutes the first systematic validation of LLM-powered, scenario-based simulation for cybersecurity literacy education—establishing its efficacy, scalability, and pedagogical viability. It introduces a novel, AI-augmented paradigm for security education grounded in authentic, adaptive interaction.

Contemporary dialogue systems typically adopt an integrated architecture combining large language models (LLMs), external tools, and databases; thus, evaluating only the underlying LLM fails to ensure end-to-end quality. Existing evaluation methods predominantly focus on single-turn analysis and lack automated, process-aware testing for full conversational trajectories. Method: We propose the first end-to-end testing framework based on non-cooperative user simulation: (1) a challenging, role-driven user simulator requiring no reference dialogues or system-internal knowledge; (2) a fine-grained error taxonomy to guide prompt optimization and enhance detection of dialogue failures and anomalies; and (3) a decoupled architecture enabling low-cost configuration and cross-system portability. Contribution/Results: Experiments demonstrate substantial improvements in defect detection rates, with strong generalizability, scalability, and robustness across diverse dialogue systems and evaluation settings.

This work addresses the limitations of existing user simulation methods, which often fail to capture the complexity and communicative friction inherent in real human–agent interactions, leading to overly optimistic evaluation results. The authors propose Realsim, a novel evaluation framework that systematically compares real and simulated multi-turn dialogues across eight dimensions—including communicative functions, user states, and surface linguistic forms—from a distributional perspective. To support this analysis, they introduce a dataset of 1,000 real conversations spanning 16 domains. Through multidimensional distributional comparisons, newly designed evaluation metrics, and empirical analysis, the study reveals that current simulators generally struggle to reproduce communicative friction and exhibit significant performance disparities across domains, thereby underscoring the need for domain-adapted user simulators.

Existing dialogue system evaluation often overlooks real-world complexities—including real-time response requirements, multi-party interaction, and long-range contextual dependencies. Method: We propose DialSim, the first dialogue simulator supporting real-time constraints, dynamic multi-role modeling, and uncertainty awareness; it enables LLMs to improvise as TV drama characters in time-limited, multi-turn, multi-party dialogues grounded in long-range context while explicitly identifying knowledge boundaries. Concurrently, we introduce LongDialQA—a benchmark dataset enabling robust evaluation under dynamically randomized questioning. Our methodology integrates role-driven simulation design, context-aware response generation, real-time latency monitoring, and a structured evaluation framework. Contribution/Results: Comprehensive experiments expose critical bottlenecks in state-of-the-art models regarding temporal constraint adherence, role tracking fidelity, and knowledge boundary calibration. DialSim establishes a reproducible, quantitative benchmark for assessing long-horizon understanding and reasoning capabilities in interactive dialogue settings.

This work addresses the challenge of effectively evaluating user simulators employed to train collaborative large language model (LLM) assistants. It proposes using downstream task utility derived from real human interactions as the primary evaluation criterion and systematically compares role-playing prompts against simulators fine-tuned on authentic dialogues from WildChat. LLM assistants trained via reinforcement learning with these simulators are evaluated on the WildBench benchmark and through large-scale user studies. Results demonstrate that assistants trained with fine-tuned simulators achieve a 58% win rate in real-user evaluations, significantly outperforming those trained with role-playing methods (51%) and exhibiting stronger generalization capabilities. These findings validate the efficacy of data-driven user simulators in enhancing the collaborative performance of LLMs.

This work addresses the underexplored Sim2Real gap between large language model (LLM)-based user simulators and actual human behavior in interactive tasks. We propose the User-Sim Index (USI), a novel metric for systematically evaluating simulation fidelity through a large-scale controlled study involving 31 LLM-based simulators and 451 real users across 165 tasks under the full τ-bench protocol. Through multidimensional behavioral analysis, rule-based reward mechanisms, and comparisons across model families, we quantitatively reveal systematic biases in LLM simulators: they consistently exhibit excessive cooperativeness, overly positive feedback, and limited stylistic diversity, leading to inflated agent success rates. Notably, improvements in general model capabilities do not necessarily translate to higher simulation fidelity. Our findings establish USI as a new standard for assessing user simulation realism and underscore the critical need for human-in-the-loop validation.

This work addresses the limitation of large language models (LLMs) in authentically simulating human-like conversational behaviors, particularly inconsistent and non-collaborative phenomena such as misunderstandings and interruptions. To this end, the authors propose CoCoEval, a novel evaluation framework that systematically assesses LLMs across ten fine-grained categories of non-collaborative behavior using an LLM-as-a-Judge approach in diverse dialogue settings—including academic discussions, business meetings, government proceedings, and debates. The study reveals that off-the-shelf LLMs significantly underestimate the prevalence of such behaviors, prompt engineering yields inconsistent improvements, and supervised fine-tuning often leads to over-generation of specific behaviors like repetition. These findings highlight fundamental challenges in modeling and controlling complex social interaction dynamics with current LLMs.

This work addresses the limitations of traditional conversational recommender systems, which struggle to accurately model complex user preferences due to sparse dialogue histories and single-turn recommendation paradigms. It also tackles the issue of existing large language model–based user simulators that, lacking explicit preference labels, suffer from accumulated feedback bias and degraded generalization. To overcome these challenges, the paper proposes the SMTPO framework, which innovatively integrates multi-task supervised fine-tuning with reinforcement learning to enhance simulated feedback quality without requiring explicit preference annotations. Notably, it introduces a fine-grained reward mechanism that guides the recommender to progressively align with real user preferences over multi-turn interactions. Extensive experiments on multiple public datasets demonstrate that the proposed approach significantly improves both recommendation accuracy and robustness, confirming its effectiveness and transferability.