ReMiX-MAE: Learning Missing-Channel Cross-Modal Representations from RGB-Only Clinical Facial Videos for Sympathetic-Mediated Pain Assessment

2026-08-03

AI Summary

This work addresses the challenges of automatic clinical pain assessment, which is hindered by sparse annotations and subtle pain-related cues in RGB signals, while more informative modalities such as thermal and depth imaging are impractical for routine deployment. To overcome this, we propose ReMiX-MAE, a self-supervised multimodal masked autoencoder framework that learns transferable facial representations from synchronized RGB, thermal, and depth video streams, explicitly modeling robustness to missing modalities and enabling inference using RGB alone. We introduce the first longitudinal dataset for sympathetically maintained pain (SMP), capturing trajectories before and after treatment, and demonstrate the first successful cross-modal representation learning pipeline from full multimodal pretraining to single-RGB deployment. Experiments show that ReMiX-MAE significantly outperforms RGB-only baselines, with pseudo-multimodal features further boosting performance on a five-class pain recognition task and exhibiting superior transfer robustness and label efficiency on external cohorts.

Abstract

Automated pain assessment in real clinics is limited by scarce clinically grounded facial video data with weak labels (often sequence-level self-report) and by the fact that pain cues can be subtle or near-neutral in RGB, while thermal and depth signals are informative yet impractical to deploy routinely. To address these challenges, we propose ReMiX-MAE (Reconstructing Missing Channel Cross-Modal Masked Autoencoder), a self-supervised multimodal masked pretraining framework that learns transferable facial representations from synchronized RGB, thermal, and depth videos and explicitly trains robustness to missing modalities, enabling RGB-only deployment. To fill the gap of clinically grounded facial pain data with video-level self-report and longitudinal treatment trajectories, we collect the Sympathetic Mediated Pain (SMP) dataset with paired pre- and post-recordings across multiple visits. Under RGB-only deployment, we evaluate ReMiX-MAE using both direct feature extraction and pseudo-multimodal features decoded from RGB. ReMiX-MAE consistently outperforms an RGB-only masked autoencoder baseline on SMP, with pseudo-multimodal features providing additional gains in the challenging five-class setting. Across external datasets, ReMiX-MAE further shows more robust and label-efficient transfer than RGB-only baselines, highlighting its advantage in data-limited clinical settings.

Problem

• pain assessment
• missing modalities
• clinical facial videos
• weak labels
• multimodal representation

Innovation

• ReMiX-MAE
• missing-modality robustness
• self-supervised multimodal learning
• pseudo-multimodal reconstruction
• clinical pain assessment