Towards Fast Domain Adaptation and Fine-Grained User Simulation for Evaluating Conversational Recommender Systems

Unknown Date

AI Summary

This work addresses the limitations of existing large language model (LLM)-based user simulators in evaluating conversational recommender systems, which often suffer from poor domain adaptability, coarse-grained user modeling, and insufficient evaluation validity. To overcome these challenges, the authors propose AdaptSim, a novel framework that enables rapid cross-domain adaptation through automated prompt generation and an open action space. AdaptSim further incorporates a "think-before-respond" strategy to exert fine-grained control over user language styles, thereby generating more realistic and diverse simulated interactions. Additionally, the authors introduce a breadth-first search–based turn-level pairwise comparison evaluation framework to comprehensively assess system performance and robustness. Experimental results across three domains and four LLMs demonstrate that AdaptSim consistently produces highly realistic dialogues and significantly enhances the effectiveness and reliability of system evaluation.

Abstract

Conversational Recommender Systems (CRSs) enhance user experience through multi-turn interactions, yet evaluating their performance remains challenging. While Large Language Model (LLM) based user simulators are effective, they suffer from three key limitations: (1) Lack of Domain Adaptability: Reliance on fixed prompts and predefined action spaces hinders transfer to novel domains; (2) Limited User Modeling: Inability to accurately replicate subtle linguistic styles and dynamic preferences; (3) Insufficient Evaluation Validity: Existing simulators fail to adequately assess fundamental capabilities and system robustness. To overcome these, we propose AdaptSim, an Adaptive domain and automatic prompt tuning User Simulator. AdaptSim offers an efficient framework for evaluating CRSs by enabling realistic behavior modeling and diverse style generation. It leverages automatic prompt generation and an open action mechanism to reduce manual effort and improve cross-domain flexibility. For response generation, we employ controlled text generation with a "think-then-respond" strategy for fine-grained control over language style. For CRS evaluation, AdaptSim incorporates a novel Breadth-First Search (BFS)-based, turn-level pairwise comparison framework for comprehensive assessment. Extensive experiments across three domains and four LLMs demonstrate that AdaptSim generates realistic dialogues, enabling a highly effective and reliable evaluation of CRS capabilities and robustness.

Problem

• Domain Adaptation
• User Simulation
• Conversational Recommender Systems
• Evaluation Validity
• Fine-Grained Modeling

Innovation

• adaptive user simulation
• automatic prompt tuning
• fine-grained language control
• open action space
• BFS-based evaluation