IDEAL: A Multimodal Domain Adaptation Framework for EEG-Eye Emotion Recognition

2026-09-30

AI Summary

This study addresses the cross-subject variability in multimodal physiological signals and the limitation of existing single-modal domain adaptation methods that overlook data-level distribution discrepancies. To this end, we propose IDEAL, a novel framework for EEG-eye movement fusion-based emotion recognition. Specifically, IDEAL introduces a pioneering multi-model collaborative selection mechanism to bridge data distribution gaps, integrating instance-level curriculum expansion with feature-level hierarchical adversarial alignment. Furthermore, an angular contrastive constraint is incorporated to optimize multimodal representation learning. Extensive experiments demonstrate that the proposed method significantly outperforms state-of-the-art approaches across four benchmark datasets. The source code has been made publicly available.

Abstract

Electroencephalography (EEG) emotion recognition serves as a pivotal interface for human-computer interaction, yet the physiological variability across individuals complicates the already challenging task of fusing heterogeneous physiological signals (e.g., EEG and eye movements). However, most prevalent domain adaptation paradigms are tailored for unimodal scenarios, failing to address the heterogeneity of multimodal signals. Furthermore, they predominantly rely on feature-level alignment, overlooking the fundamental data-level discrepancy, which risks compromising fine-grained discriminative information during aggressive adaptation. To bridge these coupled gaps, we propose Instance-based Domain Expansion and Adversarial Learning (IDEAL), a unified framework that synergizes instance-level curriculum expansion with feature-level hierarchical adversarial alignment. IDEAL first introduces a multi-model collaborative screening mechanism, which propagates high-confidence target samples to explicitly bridge the distributional gap at the data level via a quantity-quality equilibrium strategy. We provide a theoretical analysis that this instance expansion strategy strictly tightens the upper bound of the target risk. Subsequently, a hierarchical adversarial network, augmented with the angular-contrastive constraints, progressively aligns representations from low-level statistics to high-level semantics while preserving class separability. Extensive experiments on four benchmark datasets demonstrate that IDEAL significantly outperforms state-of-the-art methods. To facilitate reproducibility and future research, our source code is publicly available at https://github.com/WY-BCI-Club/IDEAL.

Problem

• Multimodal Domain Adaptation
• EEG Emotion Recognition
• Cross-subject Variability
• Heterogeneous Signal Fusion

Innovation

• Multimodal Domain Adaptation
• Instance Expansion
• Hierarchical Adversarial Learning
• EEG-Eye Emotion Recognition
• Angular-Contrastive Constraints