🤖 AI Summary
Conventional graphics pipelines impose severe memory and performance bottlenecks in backward gradient computation for 3D Gaussian Splatting (3DGS).
Method: We propose the first differentiable hardware rasterization scheme tailored for 3DGS, featuring a programmable hybrid blending rasterizer that supports gradient propagation. Our architecture integrates quad-level and subgroup-level gradient reduction, employs tile-free splat processing, optimizes atomic operations, and introduces subgroup-level parallel gradient computation. We further conduct the first systematic evaluation of mixed-precision rendering using FP16 and UNORM16, identifying their optimal trade-off between accuracy and speed.
Contribution/Results: Experiments show our backward rasterizer achieves >10× speedup over naive atomic-based methods and 3× over classical tile-based approaches. End-to-end training accelerates by 3.07×, while memory overhead amounts to only 2.67% of splat sorting memory consumption.
📝 Abstract
Recent works demonstrate the advantages of hardware rasterization for 3D Gaussian Splatting (3DGS) in forward-pass rendering through fast GPU-optimized graphics and fixed memory footprint. However, extending these benefits to backward-pass gradient computation remains challenging due to graphics pipeline constraints. We present a differentiable hardware rasterizer for 3DGS that overcomes the memory and performance limitations of tile-based software rasterization. Our solution employs programmable blending for per-pixel gradient computation combined with a hybrid gradient reduction strategy (quad-level + subgroup) in fragment shaders, achieving over 10x faster backward rasterization versus naive atomic operations and 3x speedup over the canonical tile-based rasterizer. Systematic evaluation reveals 16-bit render targets (float16 and unorm16) as the optimal accuracy-efficiency trade-off, achieving higher gradient accuracy among mixed-precision rendering formats with execution speeds second only to unorm8, while float32 texture incurs severe forward pass performance degradation due to suboptimal hardware optimizations. Our method with float16 formats demonstrates 3.07x acceleration in full pipeline execution (forward + backward passes) on RTX4080 GPUs with the MipNeRF dataset, outperforming the baseline tile-based renderer while preserving hardware rasterization's memory efficiency advantages -- incurring merely 2.67% of the memory overhead required for splat sorting operations. This work presents a unified differentiable hardware rasterization method that simultaneously optimizes runtime and memory usage for 3DGS, making it particularly suitable for resource-constrained devices with limited memory capacity.