Joint-Embedding Predictive Architecture for Sensor-based Activity Recognition

2026-07-17

AI Summary

This work addresses the heavy reliance on large-scale labeled data in sensor-based human activity recognition by proposing a self-supervised Joint Embedding Predictive Architecture (HAR-JEPA). The method learns robust and generalizable representations by jointly modeling fine-grained intra-window and long-range inter-window temporal dynamics. It introduces an encoder that effectively captures both local and global temporal structures and enhances the VICReg objective with a lightweight norm regularization term to stabilize pretraining and prevent representational collapse. Experimental results on two continuous activity recognition benchmarks demonstrate that the proposed approach substantially reduces annotation costs and achieves superior generalization—particularly on high-variance transitional activities such as sit-to-stand movements—outperforming supervised baselines.

Abstract

Sensor-based human activity recognition (HAR) has achieved significant progressed in fully supervised learning settings. However, these supervised learning models rely on large amount of labeled data, which require labor-intensive collection and meticulous annotation. To address these challenges, this paper proposes a Joint Embedding Predictive Architecture framework tailored for sensor-based HAR, designed to learn robust and generalizable representations from unlabeled datasets. The proposed framework features an encoder designed to explicitly model both the fine-grained local temporal representations within individual window and the long-term temporal sequence of adjacent windows. Furthermore, we introduce an improved Variance-Invariance-Covariance Regularization (VICReg) objective function that incorporates computationally lightweight norm term to stabilize the JEPA pre-training phase. This term balances variance, invariance and covariance constraints to prevent representation collapse. The proposed HAR-JEPA framework is evaluated using two benchmark continuously performed activity datasets. The results show that high-quality representations are successfully learned by the proposed framework. Furthermore, the representations learned by HAR-JEPA demonstrates superior generalization on minority, high variance transitional activities such as sit-to-stand and sit-to-lie where supervised learning tend to overfit due to limited support.

Problem

• sensor-based activity recognition
• labeled data dependency
• representation learning
• unlabeled data
• generalization

Innovation

• Joint Embedding Predictive Architecture
• self-supervised learning
• temporal representation learning
• VICReg regularization
• sensor-based HAR