Wave2Body: Rethinking mmWave Human Pose Estimation as Radar-to-Body Token Translation

2026-07-21

AI Summary

Millimeter-wave radar point clouds are inherently sparse and view-dependent, rendering 3D human pose estimation an under-constrained problem. To address this challenge, this work proposes a radar-to-body token translation framework that reformulates pose estimation as a cross-modal translation task. The approach employs a self-supervised radar tokenizer to extract perceptual features and integrates a pretrained compositional body-structure tokenizer to encode human pose priors, enabling effective disentanglement of perception, structural modeling, and alignment learning. A lightweight translator then establishes an end-to-end mapping between the two modalities. Extensive experiments demonstrate that the proposed method significantly outperforms existing approaches on both M4Human and mmBody benchmarks, achieving strong cross-domain generalization while maintaining low computational overhead.

Abstract

Millimeter-wave (mmWave) radar enables privacy-friendly human sensing, but its sparse point clouds are physical measurements of view-dependent electromagnetic reflections and only indirectly characterize body articulation. Recovering a complete 3D pose from such partial, geometry-dependent observations is therefore under-constrained. Existing methods directly regress joint coordinates from paired radar-pose data, relying on the same limited paired supervision to learn radar perception, human-body structure, and their alignment. This coupling can encourage dataset-specific shortcuts under ambiguous radar observations. We propose Wave2Body, a radar-to-body token translation framework that decouples these learning targets using a self-supervised mmWave tokenizer, a pretrained compositional body tokenizer that defines the output space, and a lightweight translator between them. Experiments on M4Human and mmBody show that Wave2Body achieves stronger cross-domain generalization than previous methods while incurring much lower computational costs for training and inference. All the code and experiment results are publicly available at https://github.com/Galaxywalk/Wave2Body.

Problem

• mmWave radar
• human pose estimation
• sparse point clouds
• 3D pose recovery
• view-dependent reflections

Innovation

• mmWave radar
• human pose estimation
• token translation
• self-supervised learning
• cross-domain generalization