Enhancing Granular Sentiment Classification with Chain-of-Thought Prompting in Large Language Models

2025-05-07

AI Summary

To address the limitation of numerical app-store ratings in capturing fine-grained sentiment expressed in user reviews, this paper proposes a Chain-of-Thought (CoT) prompting method based on large language models (LLMs), systematically introducing explicit stepwise reasoning to this task for the first time. We construct a manually annotated benchmark dataset of 2,000 Amazon app reviews and design interpretable reasoning paths to enhance model understanding of implicit sentiment intent. Controlled experiments against standard prompting demonstrate that CoT prompting significantly improves fine-grained sentiment classification accuracy from 84% to 93%, validating the critical performance gain conferred by structured reasoning. Our core contribution lies in the novel adaptation of the CoT paradigm to fine-grained sentiment classification—achieving simultaneous improvements in predictive accuracy and decision interpretability.

Technology Category

• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Machine Learning: Learning Preferences or Rankings
• Reasoning under Uncertainty: Applications

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

We explore the use of Chain-of-Thought (CoT) prompting with large language models (LLMs) to improve the accuracy of granular sentiment categorization in app store reviews. Traditional numeric and polarity-based ratings often fail to capture the nuanced sentiment embedded in user feedback. We evaluated the effectiveness of CoT prompting versus simple prompting on 2000 Amazon app reviews by comparing each method's predictions to human judgements. CoT prompting improved classification accuracy from 84% to 93% highlighting the benefit of explicit reasoning in enhancing sentiment analysis performance.

Problem

• Improving granular sentiment classification in app reviews
• Addressing limitations of numeric and polarity-based ratings
• Enhancing accuracy via Chain-of-Thought prompting in LLMs

Innovation

• Chain-of-Thought prompting enhances sentiment classification
• Explicit reasoning improves granular sentiment analysis
• CoT outperforms simple prompting in accuracy