VRSplat: Fast and Robust Gaussian Splatting for Virtual Reality

2025-05-15

AI Summary

This work addresses three key bottlenecks of 3D Gaussian Splatting (3DGS) in VR: motion-induced “popping” artifacts, view-inconsistent floating due to projection distortion, and severe frame-rate degradation under dense Gaussian rendering. We propose an end-to-end optimization framework featuring a novel synergistic integration of Mini-Splatting, StopThePop, and Optimal Projection. A single-threaded, foveated rasterizer is designed to efficiently handle both foveal and peripheral regions in unified rendering. Further, we incorporate depth-aware Gaussian fine-tuning, foveation-adaptive GPU rasterization, and multi-stage joint parameter optimization. The system achieves stable ≥72 FPS on consumer-grade HMDs, fully eliminating head-motion artifacts and stereo floating. A user study (n=25) demonstrates statistically significant perceptual superiority over baseline methods across immersion, visual comfort, and geometric stability metrics.

Technology Category

• Computer Vision: 3D Computer Vision
• Search and Optimization: Sampling/Simulation-based Search
• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Virtualization and resource management in Web systems and infrastructures
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

3D Gaussian Splatting (3DGS) has rapidly become a leading technique for novel-view synthesis, providing exceptional performance through efficient software-based GPU rasterization. Its versatility enables real-time applications, including on mobile and lower-powered devices. However, 3DGS faces key challenges in virtual reality (VR): (1) temporal artifacts, such as popping during head movements, (2) projection-based distortions that result in disturbing and view-inconsistent floaters, and (3) reduced framerates when rendering large numbers of Gaussians, falling below the critical threshold for VR. Compared to desktop environments, these issues are drastically amplified by large field-of-view, constant head movements, and high resolution of head-mounted displays (HMDs). In this work, we introduce VRSplat: we combine and extend several recent advancements in 3DGS to address challenges of VR holistically. We show how the ideas of Mini-Splatting, StopThePop, and Optimal Projection can complement each other, by modifying the individual techniques and core 3DGS rasterizer. Additionally, we propose an efficient foveated rasterizer that handles focus and peripheral areas in a single GPU launch, avoiding redundant computations and improving GPU utilization. Our method also incorporates a fine-tuning step that optimizes Gaussian parameters based on StopThePop depth evaluations and Optimal Projection. We validate our method through a controlled user study with 25 participants, showing a strong preference for VRSplat over other configurations of Mini-Splatting. VRSplat is the first, systematically evaluated 3DGS approach capable of supporting modern VR applications, achieving 72+ FPS while eliminating popping and stereo-disrupting floaters.

Problem

• Addressing temporal artifacts in VR with 3D Gaussian Splatting
• Reducing projection-based distortions for view-consistent rendering
• Improving framerates for large Gaussian counts in VR

Innovation

• Combines Mini-Splatting, StopThePop, Optimal Projection
• Introduces efficient foveated rasterizer for VR
• Optimizes Gaussian parameters via fine-tuning step