🤖 AI Summary
This study addresses the limitations of traditional sentiment analysis, which relies solely on customer tone and often fails to accurately capture true satisfaction. The authors propose a novel large language model–based approach that directly extracts structured indicators of customer satisfaction and underlying reasons—such as “tolerated friction”—from over 70,000 customer service dialogues. Leveraging GPT-5.4 for multidimensional annotation and validating labels against actual customer ratings, the method achieves a correlation of 0.47 between predicted satisfaction and ground-truth scores, significantly outperforming conventional sentiment analysis (r = 0.36). It also demonstrates a lower false-positive rate and reveals a notable discrepancy: in 44% of conversations, customer tone diverges from actual satisfaction, underscoring the approach’s superior discriminative capability beyond surface-level sentiment cues.
📝 Abstract
Most companies read their customer support data at scale using sentiment analysis, which measures how customers sound rather than whether they were satisfied with the result. We tested a richer alternative on 70,450 support conversations from a leading online fundraising platform: alongside tone, we used GPT-5.4 to estimate each customer's satisfaction and to flag whether they reported a concrete problem, then validated all three readings against the 1-to-5 ratings customers left on the conversations they rated. The satisfaction estimate tracked those ratings far better than sentiment did, correlating at 0.47 against 0.36 and flagging unhappy customers with far fewer false alarms. The structured read also sees what sentiment cannot: tone and satisfaction disagree in 44% of conversations, a single "Neutral" label hides everything from quietly satisfied customers to ones who quietly gave up, and the largest group of all is "tolerated friction," customers who are satisfied but still reporting a fixable problem, a standing issue that no sentiment-based dashboard can surface. The broader finding is that LLM-based annotation can capture far more than the tonality of a customer's language, offering strong potential for new business metrics grounded instead in the customer's state (whether they were satisfied) and the cause of their problem extracted directly from the raw textual data of interactions and feedback.