mmPred: Radar-based Human Motion Prediction in the Dark

2025-11-29

AI Summary

To address the limitations of RGB-D approaches—namely, sensitivity to illumination and privacy leakage—and the signal degradation issues in mmWave radar (e.g., temporal inconsistency and joint misdetection caused by specular reflection and multipath effects), this paper introduces radar modality to human motion prediction (HMP) for the first time. We propose the first diffusion-based generative framework tailored for radar signals, featuring: (1) a dual-domain historical motion representation module that jointly models temporal pose refinement and frequency-domain dominant motion; and (2) a global skeletal relation Transformer that explicitly encodes topological and cooperative joint relationships to mitigate structural distortion under radar noise. Evaluated on mmBody and mm-Fi datasets, our method achieves 8.6% and 22% improvements in prediction accuracy over state-of-the-art methods, respectively, significantly enhancing robustness and practicality in dark and privacy-sensitive scenarios.

Technology Category

• Computer Vision: Motion & Tracking
• Machine Learning: Large Multimodal Models (LMMs)
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Existing Human Motion Prediction (HMP) methods based on RGB-D cameras are sensitive to lighting conditions and raise privacy concerns, limiting their real-world applications such as firefighting and healthcare. Motivated by the robustness and privacy-preserving nature of millimeter-wave (mmWave) radar, this work introduces radar as a novel sensing modality for HMP, for the first time. Nevertheless, radar signals often suffer from specular reflections and multipath effects, resulting in noisy and temporally inconsistent measurements, such as body-part miss-detection. To address these radar-specific artifacts, we propose mmPred, the first diffusion-based framework tailored for radar-based HMP. mmPred introduces a dual-domain historical motion representation to guide the generation process, combining a Time-domain Pose Refinement (TPR) branch for learning fine-grained details and a Frequency-domain Dominant Motion (FDM) branch for capturing global motion trends and suppressing frame-level inconsistency. Furthermore, we design a Global Skeleton-relational Transformer (GST) as the diffusion backbone to model global inter-joint cooperation, enabling corrupted joints to dynamically aggregate information from others. Extensive experiments show that mmPred achieves state-of-the-art performance, outperforming existing methods by 8.6% on mmBody and 22% on mm-Fi.

Problem

• Develops a radar-based human motion prediction system for low-light conditions.
• Addresses radar signal noise and inconsistency via a diffusion-based framework.
• Enhances prediction accuracy by modeling joint cooperation and motion trends.

Innovation

• Uses millimeter-wave radar for human motion prediction
• Introduces dual-domain historical motion representation
• Employs diffusion-based framework with global skeleton transformer