Two-Stage Multimodal Framework for Emotion Mimicry Intensity Prediction

2026-05-20

AI Summary

This work addresses the prediction of mimicry intensity for six emotions—admiration, amusement, determination, empathic pain, excitement, and joy—in real-world multimodal videos. To this end, the authors propose a two-stage multimodal fusion framework: first, modality-specific encoders for text, audio, visual, and optionally motion cues are trained independently; then, their representations are integrated via a lightweight regressor. The approach incorporates a modality dropout mechanism and a controlled fine-tuning strategy to enable efficient and robust cross-modal fusion while preserving modality independence. Evaluated on the Hume-ABAW10 challenge, the method achieved third place, yielding an average Pearson correlation coefficient of 0.57 on the test set and a peak score of 0.4722 on the validation set, thereby demonstrating the effectiveness and reproducibility of the proposed fusion strategy.

Abstract

We present our submission to the Hume-ABAW10 Emotional Mimicry Intensity (EMI) Challenge, which aims to predict six continuous emotion intensity dimensions: Admiration, Amusement, Determination, Empathic Pain, Excitement, and Joy, from in-the-wild multimodal video clips. We propose a staged multimodal framework that combines textual, acoustic, and visual representations, with an optional motion branch. Our approach first trains modality-specific encoders independently and then fuses their learned representations through a lightweight regressor with modality dropout and controlled encoder adaptation. Across our submitted systems, the best validation performance is obtained by the text--audio--vision--motion fusion model under the expanded 4:1 split, achieving an average Pearson correlation of 0.4722. Although the motion branch yields only very slight gains, its behavior can be interesting to study. Our team was placed third in the EMI challenge, achieving an average Pearson correlation of 0.57 for the test set. Overall, we provide a practical and reproducible baseline for EMI prediction.

Problem

• Emotion Mimicry Intensity
• Multimodal Video
• Continuous Emotion Prediction
• In-the-wild Data
• Emotion Recognition

Innovation

• two-stage multimodal framework
• modality-specific encoders
• modality dropout
• controlled encoder adaptation
• emotion mimicry intensity prediction