HuGDiffusion: Generalizable Single-Image Human Rendering via 3D Gaussian Diffusion

2025-01-25

AI Summary

Generating high-fidelity, multi-view-consistent, and photorealistic 3D human models from a single image remains a challenging ill-posed problem. To address this, we propose a human-prior-guided, multi-stage 3D Gaussian attribute diffusion framework. Our method integrates human-centered feature extraction, conditional diffusion modeling, and staged generation of Gaussian attributes—namely 3D positions, opacities, and spherical harmonic coefficients. Crucially, we introduce an attribute-level proxy ground-truth supervision strategy, which effectively alleviates the generalization bottleneck in novel view synthesis (NVS) under unknown camera poses. Evaluated on single-image input, our approach significantly outperforms existing state-of-the-art methods, achieving consistent improvements in PSNR, LPIPS, and perceptual quality. The resulting model enables arbitrary-view rendering and robust pose-conditioned 3D reconstruction.

Technology Category

Computer Vision: Diffusion Models for VisionHumans and AI: Human-Aware Planning and Behavior PredictionCognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

User Modeling, Personalization and Recommendation: Fairness-aware retrieval and rankingEconomics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applicationsSemantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

We present HuGDiffusion, a generalizable 3D Gaussian splatting (3DGS) learning pipeline to achieve novel view synthesis (NVS) of human characters from single-view input images. Existing approaches typically require monocular videos or calibrated multi-view images as inputs, whose applicability could be weakened in real-world scenarios with arbitrary and/or unknown camera poses. In this paper, we aim to generate the set of 3DGS attributes via a diffusion-based framework conditioned on human priors extracted from a single image. Specifically, we begin with carefully integrated human-centric feature extraction procedures to deduce informative conditioning signals. Based on our empirical observations that jointly learning the whole 3DGS attributes is challenging to optimize, we design a multi-stage generation strategy to obtain different types of 3DGS attributes. To facilitate the training process, we investigate constructing proxy ground-truth 3D Gaussian attributes as high-quality attribute-level supervision signals. Through extensive experiments, our HuGDiffusion shows significant performance improvements over the state-of-the-art methods. Our code will be made publicly available.

Problem

• 3D human model generation
• single image input
• multi-angle rendering

Innovation

• HuGDiffusion
• 3DModeling
• HighFidelity