Contrastive Learning for Multimodal Human Activity Recognition with Limited Labeled Data

2026-04-25

AI Summary

This work addresses the challenges of strong modality heterogeneity and scarce labeled data in multimodal human activity recognition by proposing the CLMM framework, which employs a two-stage training strategy. In the first stage, a CNN-DiffTransformer encoder extracts cross-modal shared features enhanced by a hard positive sample weighting mechanism. The second stage leverages a dual-branch architecture to capture modality-specific characteristics, integrating shared and unique features through a primary-auxiliary collaborative training scheme. Innovatively combining contrastive learning with this two-stage paradigm, CLMM effectively models both cross-modal commonalities and modality-specific traits under limited supervision. Experimental results demonstrate that CLMM significantly outperforms state-of-the-art methods on three public benchmarks, achieving notable improvements in both accuracy and convergence speed.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Human activity recognition serves as the foundation for various emerging applications. In recent years, researchers have used collaborative sensing of multi-source sensors to capture complex and dynamic human activities. However, multimodal human activity sensing typically encounters highly heterogeneous data across modalities and label scarcity, resulting in an application gap between existing solutions and real-world needs. In this paper, we propose CLMM, a general contrastive learning framework for human activity recognition that achieves effective multimodal recognition with limited labeled data. CLMM employs a novel two-stage training strategy. In the first stage, CLMM employs a CNN-DiffTransformer encoder to capture cross-modal shared information by extracting local and global features. Meanwhile, a hard-positive samples weighting algorithm enhances gradient propagation to reinforce shared learning. In the second stage, a dual-branch architecture combining quality-guided attention and bidirectional gated units captures modality-specific information, while a primary-auxiliary collaborative training strategy fuses both shared and modality-specific information. Experimental results on three public datasets demonstrate that CLMM significantly improves state-of-the-art baselines in both recognition accuracy and convergence performance.

Problem

• multimodal human activity recognition
• limited labeled data
• heterogeneous data
• label scarcity

Innovation

• contrastive learning
• multimodal fusion
• limited labeled data
• two-stage training
• CNN-DiffTransformer