Problem
Research questions and friction points this paper is trying to address.
Multimodal Emotion Recognition
Cross-subject Domain Adaptation
Joint Distribution Collapse
Missing Modality
EEG-Eye Tracking
Aligning the Incomplete: Joint Distribution Calibration for Multimodal EEG-Eye Emotion Recognition
🤖 AI Summary
This study addresses the joint distribution collapse in multimodal emotion recognition caused by EEG domain shifts and missing eye-tracking data. To tackle this, we propose GUARD, a framework that unifies domain adaptation and missing modality imputation within a single architecture. Specifically, GUARD employs gradient-weighted objectives to guide generative modules in recovering asymmetric distributions, integrates cycle-consistent flows with unsupervised calibration techniques, and leverages perceptual losses to prioritize the preservation of emotion-discriminative semantics. Experimental results demonstrate that GUARD significantly outperforms existing state-of-the-art methods. Notably, it maintains high discriminative performance even when auxiliary modalities are entirely unavailable, thereby achieving robust cross-subject emotion recognition under challenging real-world conditions.
📝 Abstract
The success of cross-subject multimodal emotion recognition hinges on maintaining the consistency of the joint data distribution across individuals. However, real-world deployment frequently triggers the \emph{asymmetric joint distribution collapse}: EEG signals suffer from severe cross-subject distribution shifts, while eye movements sensors are susceptible to packet loss and tracking failures. Existing methods treat domain adaptation and missing-modality imputation as disjoint tasks. Consequently, they fail to resolve the compounded errors when both degradations co-occur, either propagating domain shifts through imputed signals or destroying the joint decision boundary. To tackle this unified challenge, we propose GUARD (\textbf{G}radient-guided \textbf{U}nsupervised \textbf{A}symmetric \textbf{R}ecovery of \textbf{D}istributions). First, GUARD establishes a reliable anchor manifold in the source domain by employing a theoretically grounded gradient-weighted objective, which forces the robust EEG modality to preemptively entangle task-discriminative ocular features. Next, to structurally recover the collapsed joint distribution, we constrain a generative module with downstream perceptual losses, prioritizing emotion-discriminative semantics over mere signal fidelity. Finally, we formulate target-domain adaptation as an ill-posed inverse problem. By driving a cycle-consistent flow, we achieve unsupervised calibration of the recovered joint distribution directly on the target-domain manifold. Extensive experiments demonstrate that GUARD significantly outperforms state-of-the-art methods, maintaining resilient discriminative performance even under complete auxiliary modality failure. Our code and models are made publicly available to ensure complete reproducibility.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Joint Distribution Calibration
Multimodal Emotion Recognition
Domain Adaptation
Missing-Modality Imputation
Cycle-Consistent Flow
🔎 Similar Papers
No similar papers found.
