VRGaussianAvatar: Integrating 3D Gaussian Avatars into VR

2026-02-02

AI Summary

This study addresses the challenge of achieving real-time, high-fidelity full-body 3D avatar rendering using only head-mounted display (HMD) tracking signals. To this end, the authors propose a decoupled VR system: the front end estimates full-body poses from HMD signals via inverse kinematics and simultaneously transmits pose data along with binocular camera parameters, while the back end performs stereoscopic rendering using a 3D Gaussian Splatting (3DGS) avatar reconstructed from a single image. The core innovation lies in Binocular Batching, a technique that jointly processes left- and right-eye views to significantly reduce redundant computations and enhance rendering efficiency at high VR resolutions. User studies demonstrate that the proposed method outperforms image- or video-based mesh avatar baselines in terms of visual fidelity, body ownership, and behavioral plausibility.

Technology Category

• Computer Vision: 3D Computer Vision
• Humans and AI: Game Design — Virtual Humans, NPCs and Autonomous Characters
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Systems and Infrastructure for Web, Mobile and WoT: Virtualization and resource management in Web systems and infrastructures

Abstract

We present VRGaussianAvatar, an integrated system that enables real-time full-body 3D Gaussian Splatting (3DGS) avatars in virtual reality using only head-mounted display (HMD) tracking signals. The system adopts a parallel pipeline with a VR Frontend and a GA Backend. The VR Frontend uses inverse kinematics to estimate full-body pose and streams the resulting pose along with stereo camera parameters to the backend. The GA Backend stereoscopically renders a 3DGS avatar reconstructed from a single image. To improve stereo rendering efficiency, we introduce Binocular Batching, which jointly processes left and right eye views in a single batched pass to reduce redundant computation and support high-resolution VR displays. We evaluate VRGaussianAvatar with quantitative performance tests and a within-subject user study against image- and video-based mesh avatar baselines. Results show that VRGaussianAvatar sustains interactive VR performance and yields higher perceived appearance similarity, embodiment, and plausibility. Project page and source code are available at https://vrgaussianavatar.github.io.

Problem

• 3D Gaussian Splatting
• virtual reality
• avatar
• real-time rendering
• HMD tracking

Innovation

• 3D Gaussian Splatting
• Virtual Reality Avatars
• Binocular Batching
• Real-time Rendering
• Inverse Kinematics