EmoEUS: Uncertainty Supervision for Multimodal Emotion Recognition in Conversation

2026-07-19

AI Summary

This work addresses modality-specific uncertainty in multimodal emotion recognition in conversations (MERC), which arises from modality conflicts, noise, and missing data. To tackle this challenge, the authors propose the EmoEUS framework, which enables uncertainty-aware dynamic modality weighting by learning predictive variances. Notably, EmoEUS introduces an explicit uncertainty supervision mechanism—previously unexplored in MERC—by designing a cluster-center alignment loss that explicitly links the predicted variance to the distance between emotion-modality distribution clusters and their corresponding centers. This design enhances the robustness of multimodal fusion. Experimental results demonstrate that EmoEUS consistently outperforms state-of-the-art methods on the IEMOCAP and MELD benchmarks.

Abstract

Multimodal emotion recognition in conversation (MERC) can leverage multimodal and contextual cues to boost recognition performance. However, existing fusion approaches in MERC often ignore modality-specific uncertainty across utterances caused by conflicting cues, varying noise, and missing modality-specific signals. We propose EmoEUS, an explicit uncertainty supervision framework for MERC. EmoEUS performs uncertainty-aware multimodal fusion by dynamically weighting modalities using learned variance estimates. We also introduce an explicitly supervised loss that aligns each utterance's predicted variance with the distance between the utterance's distributional representation and its emotion- and modality-specific cluster center. Experiments on IEMOCAP and MELD show that EmoEUS consistently outperforms state-of-the-art methods.

Problem

• multimodal emotion recognition
• uncertainty
• modality fusion
• conversation
• emotion recognition

Innovation

• uncertainty supervision
• multimodal fusion
• emotion recognition in conversation
• variance estimation
• modality-specific uncertainty