Technology Category
Natural Language Processing: Question AnsweringSearch and Optimization: Evaluation and AnalysisMachine Learning: Evaluation and Analysis
Transparent Reference-free Automated Evaluation of Open-Ended User Survey Responses
🤖 AI Summary
To address the lack of effective automated methods for evaluating response quality in open-ended user surveys, this paper proposes the first unsupervised, reference-free, human-written-response-oriented, and fully interpretable two-stage evaluation framework. Stage one filters invalid responses via gibberish detection; stage two constructs an empirically grounded three-dimensional scoring scheme—capturing effort, relevance, and completeness—and leverages LLM-based semantic understanding to support multilingual (English and Korean) assessment. Crucially, the method eliminates reliance on human annotations and establishes, for the first time, a structured, reproducible automatic evaluation paradigm for human-generated survey text. Experiments demonstrate strong agreement with expert judgments (Spearman’s ρ > 0.85), significantly outperforming existing metrics, and confirm its practical utility in quality prediction and automated response screening.
Technology Category
Application Category
📝 Abstract
Open-ended survey responses provide valuable insights in marketing research, but low-quality responses not only burden researchers with manual filtering but also risk leading to misleading conclusions, underscoring the need for effective evaluation. Existing automatic evaluation methods target LLM-generated text and inadequately assess human-written responses with their distinct characteristics. To address such characteristics, we propose a two-stage evaluation framework specifically designed for human survey responses. First, gibberish filtering removes nonsensical responses. Then, three dimensions-effort, relevance, and completeness-are evaluated using LLM capabilities, grounded in empirical analysis of real-world survey data. Validation on English and Korean datasets shows that our framework not only outperforms existing metrics but also demonstrates high practical applicability for real-world applications such as response quality prediction and response rejection, showing strong correlations with expert assessment.
Problem
Research questions and friction points this paper is trying to address.
Evaluating open-ended human survey responses for marketing research quality
Filtering low-quality gibberish responses to prevent misleading conclusions
Assessing response quality across effort, relevance and completeness dimensions
Innovation
Methods, ideas, or system contributions that make the work stand out.
Two-stage framework filters gibberish responses first
Evaluates effort relevance completeness using LLM capabilities
Validated on multilingual datasets for practical survey applications
🔎 Similar Papers
No similar papers found.
