UF-AMA: A unified framework for cross-domain emotion recognition via adaptive multimodal alignment

2026-05-29

AI Summary

This study addresses the limited generalization and robustness in cross-subject and cross-session emotion recognition, which arise from individual differences, contextual variations, and inconsistent quality across multimodal data. To tackle these challenges, the authors propose a cross-modal feature fusion network that integrates Transformer encoders with multi-head cross-attention mechanisms. A novel confidence-aware sample selection strategy is introduced to dynamically partition target-domain samples, combined with a multi-granularity domain adaptation approach that jointly aligns local and global feature distributions. This framework enables cross-modal knowledge distillation and enforces global consistency. Experimental results on the SEED and SEED-IV datasets demonstrate that the proposed method achieves state-of-the-art performance in both cross-subject and cross-session settings, significantly enhancing model generalization and robustness.

Abstract

In recent years, emotion recognition based on physiological signals such as electroencephalogram (EEG) has gained considerable attention, as internal physiological data offer greater objectivity and reliability compared to external behavioral data like facial expressions. However, due to distribution shifts caused by individual and contextual differences, along with variations in sample quality across modalities, constructing a cross-domain multimodal emotion recognition model with high generalization and robustness remains a key challenge. In this study, we propose a Unified Framework with Adaptive Multimodal Alignment (UF-AMA) to address cross-subject and cross-session emotion recognition using multimodal physiological signals. First, we construct a cross-modal feature fusion network comprising Transformer encoders and multi-head cross-attention modules, enabling the deep integration of EEG signals and eye-tracking data. Subsequently, we introduce a confidence-aware screening mechanism that dynamically assesses the predictive reliability of each modality branch on target domain samples, partitions samples into different quality subsets, and accordingly applies global consistency alignment and cross-modal distillation. Finally, we propose a multi-level domain adaptation framework that jointly optimizes the marginal and conditional distributions of both local modality-specific and global fusion features, thereby reducing cross-domain distribution shifts at multiple granularities. Extensive experiments on the SEED and SEED-IV datasets demonstrate that UF-AMA achieves state-of-the-art (SOTA) performance in both cross-subject and cross-session tasks. The source code is available at: https://github.com/BetterCoderLab/UF-AMA.

Problem

• cross-domain emotion recognition
• distribution shift
• multimodal physiological signals
• generalization
• robustness

Innovation

• adaptive multimodal alignment
• cross-domain emotion recognition
• confidence-aware screening
• multi-level domain adaptation
• Transformer-based fusion