Virtual Memory for 3D Gaussian Splatting

2025-06-24

AI Summary

To address the high memory footprint and low rendering efficiency of 3D Gaussian splatting in large-scale, complex scenes, this paper proposes a virtual-memory-based streaming rendering framework. Our method pioneers the integration of virtual memory and virtual texture techniques into the Gaussian point rasterization pipeline, synergistically combining visibility-driven dynamic culling, multi-level-of-detail (LoD) control, and demand-driven streaming loading to form an end-to-end optimized rendering pipeline. The approach significantly reduces peak GPU memory consumption—by up to 76%—while enabling real-time interactive rendering of large, intricate scenes on both desktop and mobile GPUs. Rendering throughput improves by up to 2.3× over baseline methods, without compromising visual fidelity. This work establishes a scalable, memory-efficient foundation for deploying Gaussian splatting in resource-constrained and large-scene applications.

Technology Category

• Computer Vision: 3D Computer Vision
• Humans and AI: Game Design — Virtual Humans, NPCs and Autonomous Characters
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Virtualization and resource management in Web systems and infrastructures
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Security and Privacy: Large-scale security measurements

Abstract

3D Gaussian Splatting represents a breakthrough in the field of novel view synthesis. It establishes Gaussians as core rendering primitives for highly accurate real-world environment reconstruction. Recent advances have drastically increased the size of scenes that can be created. In this work, we present a method for rendering large and complex 3D Gaussian Splatting scenes using virtual memory. By leveraging well-established virtual memory and virtual texturing techniques, our approach efficiently identifies visible Gaussians and dynamically streams them to the GPU just in time for real-time rendering. Selecting only the necessary Gaussians for both storage and rendering results in reduced memory usage and effectively accelerates rendering, especially for highly complex scenes. Furthermore, we demonstrate how level of detail can be integrated into our proposed method to further enhance rendering speed for large-scale scenes. With an optimized implementation, we highlight key practical considerations and thoroughly evaluate the proposed technique and its impact on desktop and mobile devices.

Problem

• Rendering large 3D Gaussian Splatting scenes efficiently
• Reducing memory usage in complex scene reconstruction
• Enhancing real-time rendering speed for detailed environments

Innovation

• Virtual memory for dynamic Gaussian streaming
• Level of detail for enhanced rendering speed
• Optimized GPU storage for reduced memory usage