Emo Pillars: Knowledge Distillation to Support Fine-Grained Context-Aware and Context-Less Emotion Classification

2025-04-23

AI Summary

Existing emotion analysis datasets suffer from contextual sparsity and coarse-grained labeling (typically ≤6 classes), while large language models (e.g., GPT-4) exhibit over-prediction biases and high deployment costs. To address these limitations, we propose a dual-mode classification framework for 28 fine-grained emotion categories. Our method introduces a narrative corpus-driven paradigm for emotion expression generation—enabling role-centered, non-redundant contextual synthesis—and designs a lightweight, distilled BERT architecture jointly optimized for both context-aware and context-agnostic training. Leveraging Mistral-7B, we construct a data synthesis pipeline and release EmoPillars, a high-quality dataset of 400K samples (100K context-rich + 300K context-free). Extensive evaluation on GoEmotions, ISEAR, and IEMOCAP demonstrates state-of-the-art performance and significantly improved cross-domain generalization.

Technology Category

• Natural Language Processing: Generation
• Machine Learning: Multimodal Learning
• Humans and AI: Emotional Intelligence

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search

Abstract

Most datasets for sentiment analysis lack context in which an opinion was expressed, often crucial for emotion understanding, and are mainly limited by a few emotion categories. Foundation large language models (LLMs) like GPT-4 suffer from over-predicting emotions and are too resource-intensive. We design an LLM-based data synthesis pipeline and leverage a large model, Mistral-7b, for the generation of training examples for more accessible, lightweight BERT-type encoder models. We focus on enlarging the semantic diversity of examples and propose grounding the generation into a corpus of narratives to produce non-repetitive story-character-centered utterances with unique contexts over 28 emotion classes. By running 700K inferences in 450 GPU hours, we contribute with the dataset of 100K contextual and also 300K context-less examples to cover both scenarios. We use it for fine-tuning pre-trained encoders, which results in several Emo Pillars models. We show that Emo Pillars models are highly adaptive to new domains when tuned to specific tasks such as GoEmotions, ISEAR, IEMOCAP, and EmoContext, reaching the SOTA performance on the first three. We also validate our dataset, conducting statistical analysis and human evaluation, and confirm the success of our measures in utterance diversification (although less for the neutral class) and context personalization, while pointing out the need for improved handling of out-of-taxonomy labels within the pipeline.

Problem

• Lack of context in emotion classification datasets
• Over-prediction and resource intensity of large language models
• Limited semantic diversity in existing emotion datasets

Innovation

• LLM-based data synthesis pipeline
• Mistral-7b for training example generation
• Fine-tuning BERT-type encoders for SOTA