How Stylistic Similarity Shapes Preferences in Dialogue Dataset with User and Third Party Evaluations

2025-07-14

AI Summary

This study investigates the relationship between stylistic similarity and user preference in open-domain dialogue, introducing the first distinction and quantification of “subjective similarity” (user self-assessments) versus “objective similarity” (third-party human annotations). Using a newly constructed, multidimensional public dataset—containing user preference labels, subjective similarity scores, and objective similarity annotations—we conduct statistical analysis. Results show a strong positive correlation between subjective similarity and user preference (r > 0.7), whereas the correlation between subjective and objective similarity is weak (r < 0.3), indicating systematic divergence. These findings underscore the dominant role of user perception in shaping preferences, challenging the conventional paradigm that relies solely on third-party evaluations for modeling stylistic adaptation. The primary contributions are: (1) establishing subjective similarity as a highly predictive indicator of user preference, and (2) releasing the first standardized benchmark dataset enabling rigorous comparative analysis of subjective versus objective stylistic similarity assessment.

Technology Category

• Natural Language Processing: Sentiment Analysis, Stylistic Analysis, and Argument Mining
• Machine Learning: Learning Preferences or Rankings
• Knowledge Representation and Reasoning: Preferences

Application Category

• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Recent advancements in dialogue generation have broadened the scope of human-bot interactions, enabling not only contextually appropriate responses but also the analysis of human affect and sensitivity. While prior work has suggested that stylistic similarity between user and system may enhance user impressions, the distinction between subjective and objective similarity is often overlooked. To investigate this issue, we introduce a novel dataset that includes users' preferences, subjective stylistic similarity based on users' own perceptions, and objective stylistic similarity annotated by third party evaluators in open-domain dialogue settings. Analysis using the constructed dataset reveals a strong positive correlation between subjective stylistic similarity and user preference. Furthermore, our analysis suggests an important finding: users' subjective stylistic similarity differs from third party objective similarity. This underscores the importance of distinguishing between subjective and objective evaluations and understanding the distinct aspects each captures when analyzing the relationship between stylistic similarity and user preferences. The dataset presented in this paper is available online.

Problem

• Investigates how stylistic similarity affects user preferences in dialogues
• Compares subjective vs objective stylistic similarity evaluations
• Analyzes correlation between user perceptions and third-party annotations

Innovation

• Novel dataset with user and third-party evaluations
• Analyzes subjective vs objective stylistic similarity
• Strong correlation between subjective similarity and preference