Learning Emotion from Motion: Kinetic Multi-Stream Skeleton Modeling with Metadata-Conditioned Weak Label Distributions

2026-07-19

AI Summary

This work addresses the challenges in skeleton-based emotion recognition, where emotional expressions are subtle and category boundaries are inherently ambiguous. To tackle these issues, the authors propose a multi-branch fusion framework comprising three components: a 6D rotation representation branch, a part-aware kinetic multi-stream branch, and a weakly supervised label distribution learning branch conditioned on metadata. Each branch is trained independently, and their predictions are integrated at the probability level during inference. This approach effectively captures both the localized dynamic characteristics of emotional expressions and the ambiguity among emotion categories. Evaluated on the DIEM-A task using 10-fold leave-one-subject-out cross-validation, the method achieves a substantial improvement over the baseline, increasing accuracy from 0.271 to 0.366 and Macro-F1 score from 0.252 to 0.353.

Abstract

Skeleton-based emotion recognition from body motion remains challenging because emotional expressions are often characterized by subtle dynamic and relational motion cues, and hard labels may not fully capture ambiguity among related emotion categories. For the DIEM-A task in the MMAC ACII 2026 Challenge, we propose a multi-branch skeleton-based emotion recognition framework that combines a 6D rotation-based branch, a part-aware kinetic multi-stream branch, and a metadata-conditioned weak label distribution learning (LDL) branch. The branches are trained independently and fused by a probability-level ensemble at inference time. In 10-fold leave-performer-out cross-validation, the proposed framework improves Accuracy from 0.271 to 0.366 and Macro-F1 from 0.252 to 0.353 over the rotation-based baseline. Explainability ablations show that velocity and bone streams, as well as arm and leg regions, provide important cues for recognizing emotional body motion.

Problem

• skeleton-based emotion recognition
• emotional body motion
• weak label distributions
• subtle motion cues
• emotion ambiguity

Innovation

• weak label distribution learning
• kinetic multi-stream
• 6D rotation representation
• metadata-conditioned modeling
• skeleton-based emotion recognition