VSMP-IMU: Video-Grounded Semantic Motion Programs for Sensor-Aware Synthetic IMU Generation

2026-08-06

AI Summary

This work addresses the performance bottleneck in wearable human activity recognition (HAR) under low-resource, class-imbalanced, and cross-user settings caused by scarce labeled data. The authors propose a video-guided, controllable IMU signal synthesis framework that, for the first time, integrates structured semantic motion programs (SMPs) with video-based pose parsing. By decoupling activity semantics from individual-specific variations, the method extracts and enhances SMPs from videos to generate semantically accurate and diverse synthetic IMU signals, which are then adapted to target devices via domain adaptation. Evaluated on five public IMU-HAR datasets, the approach achieves an average Macro-F1 score of 78.33%, outperforming models trained solely on real data by 9.77% and surpassing the strongest synthetic baseline by 4.04%. Notably, it yields improvements of 18.54% and 19.86% under low-resource and long-tailed scenarios, respectively.

Abstract

Wearable human activity recognition (HAR) is often limited by the scarcity of labeled sensor data, especially in low-resource, class-imbalanced, and subject-generalization settings. Synthetic IMU generation can reduce this dependency and enhance HAR machine learning model's performance, but existing approaches face a trade-off without addressing all factors: video-driven methods are visually grounded but sensitive to pose-estimation errors, while text-driven methods are controllable but often weakly grounded in how activities are actually performed. We present VSMP-IMU, a video-grounded framework for controllable synthetic IMU generation based on a structured Semantic Motion Program (SMP), which separates activity-defining semantics from label-preserving variation. Given an input video, VSMP-IMU extracts and augments an SMP, uses it to synthesize motion, converts the motion into virtual IMU signals, and grounds the resulting signals to the target wearable domain. We evaluate VSMP-IMU against state-of-the-art synthetic data generation methods on five public IMU-HAR datasets under leave-one-person-out evaluation. VSMP-IMU achieves an average Macro-F1 of 78.33%, improving over real-only training by 9.77% and over the strongest prior synthetic baseline by 4.04%. In low-resource settings with reduced training data-samples, it improves over real-only training by 18.54% and over the strongest prior synthetic baselines by more than 6% on average. Under long-tail evaluation in imbalanced datasets, it improves tail-class Macro-F1 by 19.86% over Real-only training and by 4.76% over SOTA. These results show that structured video-grounded semantics provide a practical foundation for controllable, wearable-relevant synthetic sensor data generation.

Problem

• Human Activity Recognition
• Synthetic IMU Generation
• Data Scarcity
• Class Imbalance
• Sensor Data

Innovation

• Semantic Motion Program
• synthetic IMU generation
• video-grounded
• wearable HAR
• controllable synthesis