MoCA: Multi-modal Cross-masked Autoencoder for Digital Health Measurements

2025-06-02

AI Summary

Multimodal time-series data from heterogeneous wearable sensors in digital health contain rich physiological information, yet supervised learning approaches are hindered by the scarcity of labeled data in clinical settings. To address this bottleneck, we propose the first cross-modal masked autoencoding framework tailored for multimodal health signals. Built upon the Transformer architecture, our method introduces a theoretically grounded cross-modal random masking strategy that jointly models intra-temporal dependencies and inter-modal correlations, enabling effective unsupervised representation learning. Extensive experiments demonstrate that our approach significantly outperforms state-of-the-art methods on both signal reconstruction and downstream health prediction tasks—including disease risk assessment and activity recognition. To foster reproducibility and adoption, we fully open-source our implementation, pre-trained models, and standardized preprocessing tools.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

The growing prevalence of digital health technologies has led to the generation of complex multi-modal data, such as physical activity measurements simultaneously collected from various sensors of mobile and wearable devices. These data hold immense potential for advancing health studies, but current methods predominantly rely on supervised learning, requiring extensive labeled datasets that are often expensive or impractical to obtain, especially in clinical studies. To address this limitation, we propose a self-supervised learning framework called Multi-modal Cross-masked Autoencoder (MoCA) that leverages cross-modality masking and the Transformer autoencoder architecture to utilize both temporal correlations within modalities and cross-modal correlations between data streams. We also provide theoretical guarantees to support the effectiveness of the cross-modality masking scheme in MoCA. Comprehensive experiments and ablation studies demonstrate that our method outperforms existing approaches in both reconstruction and downstream tasks. We release open-source code for data processing, pre-training, and downstream tasks in the supplementary materials. This work highlights the transformative potential of self-supervised learning in digital health and multi-modal data.

Problem

• Addresses lack of labeled multi-modal health data for supervised learning
• Proposes self-supervised learning for temporal and cross-modal correlations
• Improves reconstruction and downstream tasks in digital health

Innovation

• Self-supervised multi-modal learning framework
• Cross-modality masking with Transformer autoencoder
• Utilizes temporal and cross-modal correlations