PrismAvatar: Real-time animated 3D neural head avatars on edge devices

2025-02-10

AI Summary

To address the challenge of simultaneously achieving high-quality neural radiance field (NeRF) reconstruction and real-time rendering on resource-constrained edge devices, this paper proposes a lightweight 3D neural head avatars system. Methodologically, it introduces: (1) a novel drivable hybrid implicit-explicit representation, integrating a rigged prism lattice with a 3D Morphable Model (3DMM); (2) an end-to-end distillation paradigm that transfers knowledge from deformable NeRFs to skeletal mesh + neural texture representations; and (3) a CPU-GPU co-designed hybrid rendering pipeline supporting GPU-accelerated triangle rasterization. The system achieves real-time rendering at 60 fps on mobile devices, reduces GPU memory consumption significantly, and attains reconstruction quality competitive with state-of-the-art desktop-based methods. To our knowledge, this is the first work enabling high-fidelity neural head avatars to be rendered in real time via triangle rasterization on edge hardware.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Hardware-aware ML
• Humans and AI: Game Design — Virtual Humans, NPCs and Autonomous Characters

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Personalized, context-aware and across-device search

Abstract

We present PrismAvatar: a 3D head avatar model which is designed specifically to enable real-time animation and rendering on resource-constrained edge devices, while still enjoying the benefits of neural volumetric rendering at training time. By integrating a rigged prism lattice with a 3D morphable head model, we use a hybrid rendering model to simultaneously reconstruct a mesh-based head and a deformable NeRF model for regions not represented by the 3DMM. We then distill the deformable NeRF into a rigged mesh and neural textures, which can be animated and rendered efficiently within the constraints of the traditional triangle rendering pipeline. In addition to running at 60 fps with low memory usage on mobile devices, we find that our trained models have comparable quality to state-of-the-art 3D avatar models on desktop devices.

Problem

• Real-time 3D head avatar animation
• Optimized for resource-constrained edge devices
• Integrates mesh-based and deformable NeRF models

Innovation

• real-time 3D animation
• neural volumetric rendering
• efficient mobile rendering