What About Emotions? Guiding Fine-Grained Emotion Extraction from Mobile App Reviews

2025-05-29

AI Summary

Existing sentiment analysis of mobile app reviews predominantly focuses on coarse-grained polarity (e.g., positive/negative), lacking systematic modeling of fine-grained emotions (e.g., joy, anger, fear). Method: This work adapts Plutchik’s emotion wheel to the app review domain, establishing a structured annotation schema and releasing the first high-quality, human-annotated dataset for fine-grained emotion classification. We propose a hybrid annotation framework integrating iterative human labeling, large language model (LLM)-based automated labeling, and consistency evaluation. Contribution/Results: Experiments show strong agreement between LLM and human annotations (Cohen’s κ > 0.7), substantially reducing annotation cost. However, we also identify inherent limitations of fully automated labeling in complex linguistic contexts. To our knowledge, this is the first study addressing fine-grained emotion recognition in mobile app reviews, bridging a critical gap in user affect understanding within app ecosystems and establishing a new methodological paradigm.

Technology Category

• Cognitive Modeling & Cognitive Systems: Affective Computing
• Natural Language Processing: Sentiment Analysis, Stylistic Analysis, and Argument Mining
• Machine Learning: Evaluation and Analysis

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Web Mining and Content Analysis: Sentiment analysis and opinion mining
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Opinion mining plays a vital role in analysing user feedback and extracting insights from textual data. While most research focuses on sentiment polarity (e.g., positive, negative, neutral), fine-grained emotion classification in app reviews remains underexplored. This paper addresses this gap by identifying and addressing the challenges and limitations in fine-grained emotion analysis in the context of app reviews. Our study adapts Plutchik's emotion taxonomy to app reviews by developing a structured annotation framework and dataset. Through an iterative human annotation process, we define clear annotation guidelines and document key challenges in emotion classification. Additionally, we evaluate the feasibility of automating emotion annotation using large language models, assessing their cost-effectiveness and agreement with human-labelled data. Our findings reveal that while large language models significantly reduce manual effort and maintain substantial agreement with human annotators, full automation remains challenging due to the complexity of emotional interpretation. This work contributes to opinion mining by providing structured guidelines, an annotated dataset, and insights for developing automated pipelines to capture the complexity of emotions in app reviews.

Problem

• Addressing underexplored fine-grained emotion classification in app reviews
• Developing structured annotation framework for emotion taxonomy adaptation
• Evaluating feasibility of automating emotion annotation using large language models

Innovation

• Adapts Plutchik's emotion taxonomy to app reviews
• Evaluates large language models for emotion annotation
• Provides structured guidelines and annotated dataset