ControlGS: Conditioning Neural Gaussians for Downstream-Processing-Aware XR Rendering

2026-09-25

AI Summary

This study addresses the limitation of conventional 3D Gaussian Splatting (3DGS), which neglects XR post-rendering processing and optical path variations, thereby degrading end-user visual quality. We propose the first neural Gaussian rendering framework that accounts for downstream processing by constructing an end-to-end differentiable optimization pipeline that integrates post-processing operations into the training objective. Furthermore, a conditional generative model is introduced to dynamically modulate Gaussian primitives, enabling real-time adaptation to varying display and camera parameters. Extensive evaluations across multiple datasets demonstrate that this approach significantly improves end-to-end rendering quality with minimal computational overhead, establishing an efficient new paradigm for adaptive neural rendering in XR applications.

Abstract

Extended Reality (XR) users do not directly perceive the output of a rendering engine. Instead, rendered images pass through a post-processing pipeline and the physical display-optics path before reaching the eye. Critically, the exact downstream processing can vary significantly at run time, influenced by, for instance, camera pose and display power budget. Traditional 3DGS methods either implicitly assume that this downstream pipeline preserves image quality or cannot adapt to downstream processing changes. To bridge this gap, we present ControlGS, an XR Gaussian rendering pipeline that optimizes end-to-end visual quality. ControlGS models and integrates the entire downstream processing, between the rendering output and the human eye, into the optimization objective. To adapt to downstream processing at run time, ControlGS dynamically generates Gaussian primitives conditioned upon the downstream processing parameters. Experiments show that ControlGS consistently improves end-to-end post-optics XR quality across different neural Gaussian backbones and datasets, with minimal overhead. Code is available at https://horizon-lab.org/controlgs/.

Problem

• Extended Reality (XR)
• 3D Gaussian Splatting (3DGS)
• downstream processing
• end-to-end visual quality
• rendering

Innovation

• 3D Gaussian Splatting
• Extended Reality
• Downstream Processing
• Conditional Rendering
• End-to-End Optimization