Role-Playing Evaluation for Large Language Models

2025-05-19

AI Summary

Current evaluation of large language models’ (LLMs) role-playing capabilities suffers from high manual annotation costs and significant biases in automated metrics. To address this, we propose RPEval—the first multidimensional benchmark specifically designed for role-playing evaluation—systematically defining and quantifying four core dimensions: emotional understanding, decision-making reasoning, moral alignment, and role consistency. Methodologically, RPEval constructs multi-turn dialogue tasks grounded in real-world scenarios and integrates expert annotation, adversarial testing, and consistency verification to enable dual-track assessment via automated scoring and human arbitration. The framework ensures reproducibility, extensibility, and human-AI collaborative validation. We publicly release the dataset and implementation code, and conduct baseline evaluations across mainstream LLMs. Results reveal substantial deficiencies in moral alignment and long-term role consistency—insights previously unattainable due to the absence of standardized benchmarks—thereby establishing the first community-wide evaluation standard for LLM role-playing.

Technology Category

Machine Learning: Large Multimodal Models (LMMs)Natural Language Processing: (Large) Language ModelsHumans and AI: Learning Human Values and Preferences

Application Category

Search and Retrieval-Augmented AI: Web evaluation methodologies and metricsEconomics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labelingUser Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large Language Models (LLMs) demonstrate a notable capacity for adopting personas and engaging in role-playing. However, evaluating this ability presents significant challenges, as human assessments are resource-intensive and automated evaluations can be biased. To address this, we introduce Role-Playing Eval (RPEval), a novel benchmark designed to assess LLM role-playing capabilities across four key dimensions: emotional understanding, decision-making, moral alignment, and in-character consistency. This article details the construction of RPEval and presents baseline evaluations. Our code and dataset are available at https://github.com/yelboudouri/RPEval

Problem

• Evaluating LLM role-playing ability is challenging
• Human assessments are resource-intensive and automated ones biased
• Proposes RPEval benchmark for multi-dimensional role-playing evaluation

Innovation

• Introduces Role-Playing Eval benchmark
• Assesses four key role-playing dimensions
• Provides open-source code and dataset