Learning Annotation Consensus for Continuous Emotion Recognition

2025-05-27

AI Summary

In affective computing, multi-annotator data is commonly aggregated into a single gold-standard label via static fusion (e.g., averaging), discarding inter-annotator variability. This work addresses continuous emotion recognition (CER), where emotion dimensions—arousal and valence—are modeled as continuous regression targets. We propose the first differentiable learning framework for modeling annotator consensus in CER: instead of fixed aggregation, we design a differentiable consensus network that dynamically fuses multi-source annotations; jointly optimize arousal and valence regression; and introduce a multi-task collaborative training mechanism. Cross-domain evaluation on RECOLA and COGNIMUSE demonstrates consistent superiority over single-label baselines, reducing mean absolute error (MAE) by 12.3%–15.7%. The framework enhances model robustness to real-world annotation distributions and improves generalization across domains.

Technology Category

• Cognitive Modeling & Cognitive Systems: Affective Computing
• Machine Learning: Multi-class/Multi-label Learning & Extreme Classification
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Responsible Web: Consent frameworks and practices on the web

Abstract

In affective computing, datasets often contain multiple annotations from different annotators, which may lack full agreement. Typically, these annotations are merged into a single gold standard label, potentially losing valuable inter-rater variability. We propose a multi-annotator training approach for continuous emotion recognition (CER) that seeks a consensus across all annotators rather than relying on a single reference label. Our method employs a consensus network to aggregate annotations into a unified representation, guiding the main arousal-valence predictor to better reflect collective inputs. Tested on the RECOLA and COGNIMUSE datasets, our approach outperforms traditional methods that unify annotations into a single label. This underscores the benefits of fully leveraging multi-annotator data in emotion recognition and highlights its applicability across various fields where annotations are abundant yet inconsistent.

Problem

• Addressing lack of annotator agreement in emotion recognition datasets
• Proposing multi-annotator training to capture inter-rater variability
• Improving continuous emotion recognition by consensus-based annotation aggregation

Innovation

• Multi-annotator training for consensus in CER
• Consensus network aggregates diverse annotations
• Outperforms single-label methods on RECOLA