Pose Magic: Efficient and Temporally Consistent Human Pose Estimation with a Hybrid Mamba-GCN Network

2024-08-06

arXiv.org

AI Summary

Existing Transformer-based 3D human pose estimation methods face inherent trade-offs among accuracy, computational efficiency, and local joint modeling. To address this, we propose a hybrid Mamba-GCN architecture—the first to adaptively integrate state-space models (Mamba) with graph convolutional networks (GCN), forming an attention-free, causally constrained spatiotemporal modeling framework. Our design simultaneously captures long-range temporal dependencies and preserves anatomical joint topology, while enforcing strict causality to ensure real-time inference capability. Evaluated on standard benchmarks, the method achieves new state-of-the-art performance: it reduces MPJPE by 0.9 mm, cuts FLOPs by 74.1%, and significantly improves motion consistency and generalization across varying sequence lengths.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Learning with Manifolds
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Current state-of-the-art (SOTA) methods in 3D Human Pose Estimation (HPE) are primarily based on Transformers. However, existing Transformer-based 3D HPE backbones often encounter a trade-off between accuracy and computational efficiency. To resolve the above dilemma, in this work, we leverage recent advances in state space models and utilize Mamba for high-quality and efficient long-range modeling. Nonetheless, Mamba still faces challenges in precisely exploiting local dependencies between joints. To address these issues, we propose a new attention-free hybrid spatiotemporal architecture named Hybrid Mamba-GCN (Pose Magic). This architecture introduces local enhancement with GCN by capturing relationships between neighboring joints, thus producing new representations to complement Mamba's outputs. By adaptively fusing representations from Mamba and GCN, Pose Magic demonstrates superior capability in learning the underlying 3D structure. To meet the requirements of real-time inference, we also provide a fully causal version. Extensive experiments show that Pose Magic achieves new SOTA results ($downarrow 0.9 mm$) while saving $74.1%$ FLOPs. In addition, Pose Magic exhibits optimal motion consistency and the ability to generalize to unseen sequence lengths.

Problem

• Resolves accuracy-efficiency trade-off in 3D HPE.
• Improves local joint dependency modeling with GCN.
• Ensures real-time inference and motion consistency.

Innovation

• Hybrid Mamba-GCN architecture
• Adaptive fusion of Mamba and GCN
• Fully causal real-time inference version