EgoGVAE: Ego-body Mesh Reconstruction via Guided Variational Autoencoder

2026-07-30

AI Summary

This work addresses the challenge of full-body pose estimation in head-mounted device scenarios, where the body is largely invisible to the user. To tackle this problem, the authors propose an efficient reconstruction method based on a Guided Variational Autoencoder (Guided VAE) that maps head pose into a latent space aligned with full-body poses. By leveraging a novel guidance mechanism that aligns the latent distributions of head and full-body pose networks, the model enables rapid and natural decoding of full-body meshes through a single forward sampling step—bypassing the iterative refinement required by diffusion models. Experimental results demonstrate that the proposed approach achieves over 50× faster inference than diffusion-based methods on standard benchmarks while significantly improving the quality of ego-centric body mesh reconstruction.

Abstract

We address the problem of recovering the full-body mesh from only the head pose. This task has become essential for various applications based on head-mounted devices or smart glasses. The challenge of this task lies in estimating the pose information of unobserved body parts based solely on a single joint (i.e., head) trajectory. Several studies have begun to adopt head-conditioned generative models, however, such previous methods are costly and time-consuming due to the diffusion-based iterative process. As an alternative, we propose a simple yet novel method that leverages the latent space of the guidance network, which is designed as a variational autoencoder taking full-body poses as inputs. By enforcing latent distributions of this guidance network and our head-to-motion network to be similar, latent features sampled from the 'guided' distribution, i.e., distribution learned in our head-to-motion network, can be reliably decoded for natural representations of full-body poses even only with the head pose. One important advantage of the proposed method is that one-step sampling scheme achieves remarkably fast inference (more than 50 times faster) compared to diffusion-based approaches. Experimental results on benchmark datasets show that the proposed method efficiently improves the performance of ego-body mesh reconstruction.

Problem

• ego-body mesh reconstruction
• head pose
• full-body pose estimation
• generative modeling
• latent space

Innovation

• guided variational autoencoder
• ego-body mesh reconstruction
• head-to-motion generation
• latent distribution alignment
• one-step sampling