Motion Capture is Not the Target Domain: Scaling Synthetic Data for Learning Motion Representations

2026-02-11

AI Summary

This study addresses the challenges of limited real-world data and poor transferability of synthetic data in wearable-based human activity recognition (HAR). The authors pretrain temporal models using synthetic motion data generated from motion capture systems and systematically evaluate their transfer performance across diverse downstream HAR tasks, while analyzing the impact of domain discrepancies between synthetic and real wearable signals. Findings indicate that relying solely on motion capture data yields only marginal gains due to domain mismatch; however, significant improvements in model generalization are achieved when synthetic data is combined with real data or when the scale of synthetic data is sufficiently large. This work elucidates both the potential and limitations of synthetic motion data for learning transferable HAR representations, offering a promising pathway toward low-cost, highly generalizable HAR systems.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Computer Vision: Motion & Tracking
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Web Mining and Content Analysis: Web data generation and simulation
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Synthetic data offers a compelling path to scalable pretraining when real-world data is scarce, but models pretrained on synthetic data often fail to transfer reliably to deployment settings. We study this problem in full-body human motion, where large-scale data collection is infeasible but essential for wearable-based Human Activity Recognition (HAR), and where synthetic motion can be generated from motion-capture-derived representations. We pretrain motion time-series models using such synthetic data and evaluate their transfer across diverse downstream HAR tasks. Our results show that synthetic pretraining improves generalisation when mixed with real data or scaled sufficiently. We also demonstrate that large-scale motion-capture pretraining yields only marginal gains due to domain mismatch with wearable signals, clarifying key sim-to-real challenges and the limits and opportunities of synthetic motion data for transferable HAR representations.

Problem

• synthetic data
• domain mismatch
• human activity recognition
• motion capture
• transfer learning

Innovation

• synthetic data
• motion capture
• human activity recognition
• domain mismatch
• pretraining