Tapor: 3D Hand Pose Reconstruction with Fully Passive Thermal Sensing for Around-device Interactions

2025-01-29

AI Summary

Non-contact gesture interaction in privacy-sensitive scenarios poses significant challenges due to reliance on RGB cameras, which inherently risk visual privacy leakage. Method: This paper proposes a 3D hand pose reconstruction system leveraging a low-cost, single-point thermal imaging array. We introduce the first physics-informed neural networks—thermal-to-depth and thermal-to-pose—that formulate 3D pose estimation as a thermal-map retrieval task. An heterogeneous knowledge distillation framework is further designed to achieve model lightweighting, reducing computational cost by 377×. Contribution/Results: The system enables fully passive, RGB-free, zero-privacy-leakage peripheral device control, supporting fine-grained finger tracking and multi-class gesture recognition. Evaluated in real-world settings, it achieves high-fidelity 3D reconstruction accuracy. To foster reproducibility and adoption, we publicly release a thermal imaging dataset, embedded firmware, and demonstration videos—establishing a new paradigm for privacy-preserving, edge-deployable human–computer interaction.

Technology Category

• Computer Vision: Biometrics, Face, Gesture & Pose
• Humans and AI: Interaction Techniques and Devices
• Machine Learning: Privacy

Application Category

• Security and Privacy: Data transparency and provenance
• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

This paper presents the design and implementation of Tapor, a privacy-preserving, non-contact, and fully passive sensing system for accurate and robust 3D hand pose reconstruction for around-device interaction using a single low-cost thermal array sensor. Thermal sensing using inexpensive and miniature thermal arrays emerges with an excellent utility-privacy balance, offering an imaging resolution significantly lower than cameras but far superior to RF signals like radar or WiFi. The design of Tapor, however, is challenging, mainly because the captured temperature maps are low-resolution and textureless. To overcome the challenges, we investigate the thermo-depth and thermo-pose properties and present a novel physics-inspired neural network design that learns effective 3D spatial representations of potential hand poses. We then formulate the 3D pose reconstruction problem as a distinct retrieval task, enabling precise determination of the hand pose corresponding to the input temperature map. To deploy Tapor on IoT devices, we introduce an effective heterogeneous knowledge distillation method that reduces the computation by 377x. We fully implement Tapor and conduct comprehensive experiments in various real-world scenarios. The results demonstrate the remarkable performance of Tapor, which is further illustrated by four case studies of gesture control and finger tracking. We envision Tapor to be a ubiquitous interface for around-device control and have released the dataset, software, firmware, and demo videos at https://github.com/IOT-Tapor/TAPOR.

Problem

• Non-contact Thermal Sensing
• Gesture Recognition
• 3D Hand Pose Reconstruction

Innovation

• Thermal Sensing
• Custom Neural Network
• Privacy-Preserving Gesture Recognition