mmSimPrior: Learning Simulation Priors for Data-Efficient Real-World Generalizable Radar-Based Human Motion Reconstruction

2026-07-24

AI Summary

This work addresses the challenges of mmWave radar-based human motion reconstruction in real-world scenarios, which are hindered by scarce paired data and simulation-to-reality domain gaps. The authors propose mmSimPrior, a novel framework that decouples transferable knowledge into three types of priors—signal, motion, and mapping—and integrates physics-guided domain randomization pretraining, a discrete motion codebook, and a continuous regression pathway to enable both zero-shot inference and few-shot adaptation. Evaluated with only 24 real sequences, mmSimPrior-Reg reduces MPJPE by 24.7%–39.0% across three environments, while mmSimPrior-Cls achieves an 8.5% improvement in zero-shot performance without fine-tuning. The study also establishes the first large-scale, non-overlapping test benchmark for this task.

Abstract

Millimeter-wave (mmWave) radar offers privacy-preserving and lighting-robust sensing for human motion reconstruction, but learning models that generalize across real deployments require diverse paired radar-motion data that are costly to collect. Simulation provides scalable supervision, yet models trained on clean synthetic signals transfer poorly because of multipath, clutter, response statistics, and resolution degradation. We present mmSimPrior, a simulation-pretrained framework that factorizes transferable knowledge into signal, motion, and radar-to-motion mapping priors. A multi-modal signal encoder is pretrained with a physics-informed domain-randomization curriculum that emulates propagation- and acquisition-level variations, while a joint-temporal tokenizer learns a discrete prior over plausible human motion. A shared mapping prior supports classification over a learned motion codebook for constrained zero-shot reconstruction and continuous regression for flexible limited-data adaptation. We further construct a 4.2M-frame, 31K-sequence dataset suite and introduce a No-Overlap Setting that excludes repeated complete subject-environment-location-motion configurations across adaptation and test. Experiments on mmSimPrior-Real and RT-Pose demonstrate consistent gains: with only 24 paired real sequences, mmSimPrior-Reg reduces MPJPE by 24.7% to 39.0% over the strongest baseline across the three environments, while mmSimPrior-Cls reduces zero-shot MPJPE by 8.5% without finetuning.

Problem

• mmWave radar
• human motion reconstruction
• simulation-to-real transfer
• data efficiency
• domain generalization

Innovation

• simulation prior
• domain randomization
• mmWave radar
• motion reconstruction
• zero-shot generalization