Leveraging 2D Masked Reconstruction for Domain Adaptation of 3D Pose Estimation

2025-01-14

AI Summary

Poor generalization of RGB-based 3D pose estimation in cross-domain settings—particularly under significant distribution shift between training and test data—remains a critical challenge. To address this, we propose an unsupervised domain adaptation method grounded in masked image modeling (MIM). Our approach introduces three key innovations: (1) the first incorporation of 2D foreground mask modeling into 3D pose estimation domain adaptation; (2) a foreground-centered reconstruction loss coupled with cross-layer attention regularization to enhance utilization of unlabeled target-domain data; and (3) a dual-domain joint optimization framework that integrates foreground-aware masking with self-supervised pretraining. Evaluated on multiple cross-domain benchmarks for both human and hand 3D pose estimation, our method consistently outperforms existing supervised and unsupervised domain adaptation approaches, achieving state-of-the-art performance.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Computer Vision: 3D Computer Vision
• Application Domains: Humanities & Computational Social Science

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

RGB-based 3D pose estimation methods have been successful with the development of deep learning and the emergence of high-quality 3D pose datasets. However, most existing methods do not operate well for testing images whose distribution is far from that of training data. However, most existing methods do not operate well for testing images whose distribution is far from that of training data. This problem might be alleviated by involving diverse data during training, however it is non-trivial to collect such diverse data with corresponding labels (i.e. 3D pose). In this paper, we introduced an unsupervised domain adaptation framework for 3D pose estimation that utilizes the unlabeled data in addition to labeled data via masked image modeling (MIM) framework. Foreground-centric reconstruction and attention regularization are further proposed to increase the effectiveness of unlabeled data usage. Experiments are conducted on the various datasets in human and hand pose estimation tasks, especially using the cross-domain scenario. We demonstrated the effectiveness of ours by achieving the state-of-the-art accuracy on all datasets.

Problem

• Color Information
• 3D Pose Estimation
• Generalization Performance

Innovation

• 3D Pose Estimation
• Mask Image Modeling
• Unsupervised Data Optimization