VARPose: Flexible 2D Pose Densification via Visual Autoregressive Modeling for Enhanced 3D Lifting

2026-08-03

AI Summary

This work addresses the challenge of leveraging sparse 2D human poses to improve 3D pose estimation and body mesh reconstruction. To this end, the authors propose UniSkelar, a unified autoregressive model, together with GPT, a granularity-agnostic pose encoder. By integrating a hybrid codebook, residual quantization, and multi-scale discrete representations, the framework enables cross-granularity dense 2D pose generation without requiring retraining for new pose granularities. The method outperforms existing approaches in 2D pose densification and significantly enhances the accuracy and generalization capability of downstream 3D tasks.

Abstract

Visual AutoRegressive Modeling (VAR) has excelled in natural image generation via next-scale prediction, but its use on topology-structured data like human skeletons is still unexplored. VARPose is proposed to adaptively densify 2D sparse poses, thereby enriching the anatomical information available for 3D lifting models. Our core contributions are twofold. First, we introduce a Granularity-agnostic Pose Tokenizer (GPT), which employs a single hybrid codebook and a residual quantization strategy to encode poses of varying densities into a unified, multi-scale discrete representation. Our results demonstrate the strong generalizability of this representation. By decoupling the representation from the projection, we can successfully decode novel pose granularities using a frozen codebook with a retrained decoder. Second, we propose UniSkelar, a unified autoregressive model that treats "joint density" as "scale". UniSkelar learns to predict the token sequence for the next density level in a coarse-to-fine manner, conditioned on the sparsest pose. VARPose not only outperforms state-of-the-art methods and generalizes to unseen granularities, but also confers tangible performance gains on downstream tasks, such as 3D Pose Estimation and Human Mesh Recovery, through 2D pose densification. Our code and model are available at https://github.com/BRL-SYSU/VARPose.git.

Problem

• 2D pose densification
• human skeleton
• topology-structured data
• 3D pose lifting
• pose granularity

Innovation

• Visual Autoregressive Modeling
• Pose Densification
• Granularity-agnostic Tokenization
• UniSkelar
• 3D Pose Lifting