Person Identification from Egocentric Human-Object Interactions using 3D Hand Pose

2025-09-20

AI Summary

To address the need for seamless and high-security authentication in augmented reality (AR), this paper proposes an implicit user identification method leveraging first-person-view hand–object interaction (HOIR) and 3D hand pose analysis. To characterize the structural geometry of the interaction space, we introduce the Inter-Hand Spatial Envelope (IHSE), a novel feature descriptor. We further design I²S, a lightweight multi-stage framework that jointly models spatial, frequency-domain, kinematic, directional, and IHSE features for end-to-end sequence learning. The method operates without explicit user cooperation. Evaluated on a dual-phone mechanical manipulation dataset, it achieves a mean F1-score of 97.52%, with a model size of only 3.8 MB and an inference latency of 0.1 seconds per sample—enabling real-time on-device deployment.

Technology Category

• Humans and AI: Interaction Techniques and Devices
• Computer Vision: Biometrics, Face, Gesture & Pose
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Personalized, context-aware and across-device search

Abstract

Human-Object Interaction Recognition (HOIR) and user identification play a crucial role in advancing augmented reality (AR)-based personalized assistive technologies. These systems are increasingly being deployed in high-stakes, human-centric environments such as aircraft cockpits, aerospace maintenance, and surgical procedures. This research introduces I2S (Interact2Sign), a multi stage framework designed for unobtrusive user identification through human object interaction recognition, leveraging 3D hand pose analysis in egocentric videos. I2S utilizes handcrafted features extracted from 3D hand poses and per forms sequential feature augmentation: first identifying the object class, followed by HOI recognition, and ultimately, user identification. A comprehensive feature extraction and description process was carried out for 3D hand poses, organizing the extracted features into semantically meaningful categories: Spatial, Frequency, Kinematic, Orientation, and a novel descriptor introduced in this work, the Inter-Hand Spatial Envelope (IHSE). Extensive ablation studies were conducted to determine the most effective combination of features. The optimal configuration achieved an impressive average F1-score of 97.52% for user identification, evaluated on a bimanual object manipulation dataset derived from the ARCTIC and H2O datasets. I2S demonstrates state-of-the-art performance while maintaining a lightweight model size of under 4 MB and a fast inference time of 0.1 seconds. These characteristics make the proposed framework highly suitable for real-time, on-device authentication in security-critical, AR-based systems.

Problem

• Identifying users unobtrusively from egocentric videos of human-object interactions.
• Recognizing human-object interactions using 3D hand pose analysis.
• Enabling real-time, on-device authentication for security-critical AR systems.

Innovation

• 3D hand pose analysis for egocentric videos
• Sequential feature augmentation for HOI recognition
• Novel Inter-Hand Spatial Envelope descriptor