Joint Geometry-Appearance Human Reconstruction in a Unified Latent Space via Bridge Diffusion

2026-01-01

arXiv.org

AI Summary

This work addresses the inconsistency commonly arising in existing methods for reconstructing 3D digital humans from a single RGB image, which stems from decoupled modeling of geometry and appearance. The authors propose JGA-LBD, a novel framework that unifies heterogeneous conditions—such as depth maps and SMPL parameters—into a shared 3D Gaussian representation, enabling joint modeling of geometry and appearance within a common latent space. By employing a shared sparse variational autoencoder to compress the 3D Gaussian representation and integrating a bridging diffusion mechanism with dedicated decoding modules, the method achieves high-quality reconstruction of complete 3D human bodies and novel view synthesis from partial observations. Extensive experiments demonstrate that JGA-LBD outperforms state-of-the-art approaches in both geometric fidelity and appearance quality, particularly excelling in complex in-the-wild scenarios.

Technology Category

Computer Vision: 3D Computer VisionMachine Learning: Deep Generative Models & AutoencodersHumans and AI: Game Design — Virtual Humans, NPCs and Autonomous Characters

Application Category

Economics, Online Markets and Human Computation: LLM based quality controls for crowd workSemantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactionsUser Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Achieving consistent and high-fidelity geometry and appearance reconstruction of 3D digital humans from a single RGB image is inherently a challenging task. Existing studies typically resort to decoupled pipelines for geometry estimation and appearance synthesis, often hindering unified reconstruction and causing inconsistencies. This paper introduces \textbf{JGA-LBD}, a novel framework that unifies the modeling of geometry and appearance into a joint latent representation and formulates the generation process as bridge diffusion. Observing that directly integrating heterogeneous input conditions (e.g., depth maps, SMPL models) leads to substantial training difficulties, we unify all conditions into the 3D Gaussian representations, which can be further compressed into a unified latent space through a shared sparse variational autoencoder (VAE). Subsequently, the specialized form of bridge diffusion enables to start with a partial observation of the target latent code and solely focuses on inferring the missing components. Finally, a dedicated decoding module extracts the complete 3D human geometric structure and renders novel views from the inferred latent representation. Experiments demonstrate that JGA-LBD outperforms current state-of-the-art approaches in terms of both geometry fidelity and appearance quality, including challenging in-the-wild scenarios. Our code will be made publicly available at https://github.com/haiantyz/JGA-LBD.

Problem

• 3D human reconstruction
• geometry-appearance consistency
• single-image reconstruction
• unified latent space
• digital human modeling

Innovation

• unified latent space
• bridge diffusion
• 3D Gaussian representation
• joint geometry-appearance reconstruction
• sparse VAE