PoseBH: Prototypical Multi-Dataset Training Beyond Human Pose Estimation

2025-05-23

AI Summary

To address the generalization bottleneck in multi-dataset training caused by skeletal structural heterogeneity (e.g., human vs. animal vs. full-body skeletons), this paper proposes a non-parametric keypoint prototype embedding framework coupled with cross-type self-supervised alignment. Without relying on teacher models or data augmentation, our method achieves consistent representation learning for heterogeneous keypoints via unified embedding space modeling and self-supervised alignment across skeleton types. Jointly optimized on diverse, heterogeneous datasets—including COCO-WholeBody, AP-10K, and APT-36K—it significantly improves cross-domain generalization. The approach maintains state-of-the-art performance on standard benchmarks (COCO, MPII, AIC) and successfully transfers to specialized domains: InterHand2.6M (hand pose estimation) and 3DPW (3D human pose estimation), demonstrating strong adaptability and broad applicability across anatomical and dimensional variations.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Computer Vision: Biometrics, Face, Gesture & Pose
• Intelligent Robots: Embodied AI

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search

Abstract

We study multi-dataset training (MDT) for pose estimation, where skeletal heterogeneity presents a unique challenge that existing methods have yet to address. In traditional domains, eg regression and classification, MDT typically relies on dataset merging or multi-head supervision. However, the diversity of skeleton types and limited cross-dataset supervision complicate integration in pose estimation. To address these challenges, we introduce PoseBH, a new MDT framework that tackles keypoint heterogeneity and limited supervision through two key techniques. First, we propose nonparametric keypoint prototypes that learn within a unified embedding space, enabling seamless integration across skeleton types. Second, we develop a cross-type self-supervision mechanism that aligns keypoint predictions with keypoint embedding prototypes, providing supervision without relying on teacher-student models or additional augmentations. PoseBH substantially improves generalization across whole-body and animal pose datasets, including COCO-WholeBody, AP-10K, and APT-36K, while preserving performance on standard human pose benchmarks (COCO, MPII, and AIC). Furthermore, our learned keypoint embeddings transfer effectively to hand shape estimation (InterHand2.6M) and human body shape estimation (3DPW). The code for PoseBH is available at: https://github.com/uyoung-jeong/PoseBH.

Problem

• Addressing skeletal heterogeneity in multi-dataset pose estimation
• Integrating diverse skeleton types without cross-dataset supervision
• Improving generalization across whole-body and animal pose datasets

Innovation

• Nonparametric keypoint prototypes in unified space
• Cross-type self-supervision without teacher models
• Seamless integration across diverse skeleton types