Interactive Rendering of Relightable and Animatable Gaussian Avatars

2024-07-15

IEEE Transactions on Visualization and Computer Graphics

AI Summary

To address the challenge of real-time, relightable, and animatable virtual human modeling from multi-view or monocular video, this paper proposes a Gaussian Splatting-based real-time avatar representation. The method integrates Gaussian Splatting with forward skinning binding for the first time, decoupling body material from environmental illumination. It introduces a fast shadow modeling mechanism via dense-view rasterization and combines signed distance function (SDF)-based canonical volume reconstruction, vertex attribute interpolation, and visibility-aware rasterization to enable joint editing of lighting, pose, and viewpoint at 6.9 fps. Evaluated on both synthetic and real-world datasets, our approach significantly outperforms NeRF- and ray-tracing-based methods in rendering quality while accelerating training and inference by an order of magnitude. It supports interactive relighting and animation generation without sacrificing visual fidelity.

Technology Category

• Humans and AI: Game Design — Virtual Humans, NPCs and Autonomous Characters
• Computer Vision: Computational Photography, Image & Video Synthesis
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Search and Retrieval-Augmented AI: Assisted, interactive, and conversational search
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Creating relightable and animatable avatars from multi-view or monocular videos is a challenging task for digital human creation and virtual reality applications. Previous methods rely on neural radiance fields or ray tracing, resulting in slow training and rendering processes. By utilizing Gaussian Splatting, we propose a simple and efficient method to decouple body materials and lighting from sparse-view or monocular avatar videos, so that the avatar can be rendered simultaneously under novel viewpoints, poses, and lightings at interactive frame rates (6.9 fps). Specifically, we first obtain the canonical body mesh using a signed distance function and assign attributes to each mesh vertex. The Gaussians in the canonical space then interpolate from nearby body mesh vertices to obtain the attributes. We subsequently deform the Gaussians to the posed space using forward skinning, and combine the learnable environment light with the Gaussian attributes for shading computation. To achieve fast shadow modeling, we rasterize the posed body mesh from dense viewpoints to obtain the visibility. Our approach is not only simple but also fast enough to allow interactive rendering of avatar animation under environmental light changes. Experiments demonstrate that, compared to previous works, our method can render higher quality results at a faster speed on both synthetic and real datasets.

Problem

• Creating relightable animatable avatars from sparse monocular videos
• Decoupling body materials lighting for interactive rendering
• Fast shadow modeling via rasterized visibility computation

Innovation

• Uses Gaussian Splatting for efficient rendering
• Decouples materials and lighting from videos
• Employs forward skinning for pose deformation