Problem
Research questions and friction points this paper is trying to address.
Egocentric action recognition
3D hand pose estimation
Hand type prior
Monocular video
Functional Hand Type Prior for 3D Hand Pose Estimation and Action Recognition from Egocentric View Monocular Videos
🤖 AI Summary
This study addresses the limitation of relying solely on geometric information for perceiving dynamic hand motions in first-person action recognition. To this end, it proposes a semantic interaction understanding framework based on functional hand shape priors. Specifically, this work constructs a function-oriented hand shape taxonomy, leveraging hand shape semantics as prior knowledge to enhance the network's comprehension of continuous interactions. Furthermore, it designs an action recognition architecture that integrates feature extraction, an egocentric knowledge module for jointly estimating 3D poses and hand shapes, and long-term temporal aggregation. Experimental results demonstrate that the proposed method surpasses existing state-of-the-art approaches on both the FPHA and H2O benchmarks, significantly improving first-person action recognition performance.
📝 Abstract
Current methods for egocentric view action recognition often face challenges in perceiving dynamic hand movements relying solely on geometrical or physical information. In this work, we effectively address this problem by gaining insights into the correlation between functional hand configurations and objects, which improves the detailed interpretation of real-world scenarios. To this end, we introduce a practical taxonomy of hand types based on the functioning perspective and utilize it for per-frame hand type labeling on existing datasets. We also propose a novel hand action recognition framework considering semantic details of the hand type as prior. This approach boosts the network's understanding of the continuous hand interaction throughout the action sequence. Our whole pipeline consists of three main modules: (1) Feature Extraction, (2) Egocentric Knowledge Module, which estimates 3D hand pose, object category, and hand type leveraging short-term cues, and (2) Egocentric Action Module, which aggregates per-frame knowledge, including text embeddings of hand type, over a longer time. In our extensive experiments with large-scale benchmarks, FPHA and H2O, our model outperforms current state-of-the-art methods, demonstrating its superior performance.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Functional Hand Type Prior
Egocentric Action Recognition
3D Hand Pose Estimation
Hand-Object Interaction
Temporal Aggregation
🔎 Similar Papers
No similar papers found.
