🤖 AI Summary
To address the trade-off between accuracy and efficiency in real-time monocular RGB-based hand pose estimation and mesh reconstruction, this paper proposes a lightweight and efficient framework. Our method introduces a novel joint-skeleton feature refinement mechanism with joint optimization, a feature interaction and expansion module to collaboratively model the 2D-to-3D mapping, and coordinate attention to enhance keypoint representation. It integrates a unified 2D/3D keypoint generator, multi-head self-attention, and linear vertex mapping to improve geometric consistency and inference speed. Evaluated on FreiHand, our approach achieves 72 FPS with PA-MPJPE of 6.3 mm, PA-MPVPE of 6.4 mm, F@05 = 0.756, and F@15 = 0.984—outperforming existing state-of-the-art methods in both accuracy and latency. The framework is particularly suitable for low-latency applications such as robotic dexterous manipulation.
📝 Abstract
Accurate hand pose estimation is vital in robotics, advancing dexterous manipulation in human-computer interaction. Toward this goal, this paper presents ReJSHand (which stands for Refined Joint and Skeleton Features), a cutting-edge network formulated for real-time hand pose estimation and mesh reconstruction. The proposed framework is designed to accurately predict 3D hand gestures under real-time constraints, which is essential for systems that demand agile and responsive hand motion tracking. The network's design prioritizes computational efficiency without compromising accuracy, a prerequisite for instantaneous robotic interactions. Specifically, ReJSHand comprises a 2D keypoint generator, a 3D keypoint generator, an expansion block, and a feature interaction block for meticulously reconstructing 3D hand poses from 2D imagery. In addition, the multi-head self-attention mechanism and a coordinate attention layer enhance feature representation, streamlining the creation of hand mesh vertices through sophisticated feature mapping and linear transformation. Regarding performance, comprehensive evaluations on the FreiHand dataset demonstrate ReJSHand's computational prowess. It achieves a frame rate of 72 frames per second while maintaining a PA-MPJPE (Position-Accurate Mean Per Joint Position Error) of 6.3 mm and a PA-MPVPE (Position-Accurate Mean Per Vertex Position Error) of 6.4 mm. Moreover, our model reaches scores of 0.756 for F@05 and 0.984 for F@15, surpassing modern pipelines and solidifying its position at the forefront of robotic hand pose estimators. To facilitate future studies, we provide our source code at ~url{https://github.com/daishipeng/ReJSHand}.