RF-HOI: Recognize Human-Object Interaction with Radio Frequency Signals

2026-07-31

AI Summary

This work addresses the challenges of privacy leakage and degraded performance under low-light conditions that plague existing vision-based methods for human-object interaction recognition. The authors propose the first purely radio-frequency (RF) modality framework, which fuses millimeter-wave radar and RFID signals to jointly perform action recognition and target object identification. To overcome the scarcity of real-world RF data, they develop a scalable RF data simulator that generates synthetic multimodal RF data, enabling effective few-shot fine-tuning. Experimental results demonstrate that the proposed approach significantly outperforms existing RF-based baselines, achieving performance comparable to vision-based models, while the synthetic data substantially enhances generalization in real-world scenarios.

Abstract

Recognizing Human-Object Interactions (HOI) is essential for intelligent systems, underpinning applications in virtual and augmented reality, embodied AI, and assistive robotics. However, vision-based HOI methods face challenges in privacy concerns and poor light conditions. In this work, we introduce RF-HOI, the first framework that only uses radio frequency (RF) signals for HOI recognition. A key challenge of RF-HOI is that single-modality RF sensing is insufficient to recognize both actions and the objects being interacted with. RF-HOI addresses this through a novel modality fusion that combines mmWave radar and RFID, enabling simultaneous action recognition and target identification. Another challenge is limited training data across diverse setups, which impairs the generalizability of the recognition model. To overcome this, we develop a simulator that synthesizes multimodal RF data for diverse HOIs at scale, allowing us to fine-tune with only a small amount of real-world data. Experiment results show that RF-HOI outperforms all baselines, approaching vision model performance, and that our diverse synthetic training data can significantly boost our system's performance on real-world scenarios. These results highlight the potential of multimodal RF sensing for robust and privacy-preserving HOI recognition as well as the effectiveness of our RF data synthesis.

Problem

• Human-Object Interaction
• RF sensing
• privacy-preserving
• multimodal fusion
• data scarcity

Innovation

• RF-HOI
• multimodal RF sensing
• mmWave radar
• RFID
• RF data synthesis