TRON: Tracing Rays to Orchestrate a Neural Renderer for 3D Gaussian Reconstructions

2026-06-09

AI Summary

Existing 3D Gaussian relighting methods are limited by inaccuracies in geometry, material, and light transport estimation, while purely neural rendering lacks explicit representations, hindering fine-grained editing. This work proposes a novel framework that integrates 3D Gaussian ray tracing with lightweight neural rendering: intrinsic decomposition priors from inverse rendering regularize the Gaussian field’s material properties, and ray-traced results serve as a structured 3D scaffold to guide the neural renderer. For the first time, this approach unifies explicit 3D Gaussian representations with neural rendering, achieving photorealistic results under dynamic lighting, geometry, and materials while preserving interactivity and controllability. Experiments demonstrate that our method outperforms existing techniques in realism, editability, and rendering speed, and enables practical interactive applications on real-world captured scenes for the first time.

Abstract

We introduce TRON, a rendering framework that combines 3D Gaussian ray tracing with neural rendering to enable realistic and controllable rendering of real-world 3D scenes under novel lighting, dynamic object motion, object insertion, and material editing. Prior approaches that rely solely on physically based rendering (PBR) of Gaussian representations struggle to achieve realistic relighting due to imperfections in reconstructed geometry, material estimates, and light transport estimation. At the same time, neural rendering methods often lack an explicit scene representation, limiting their ability to support interactive editing with fine-grained manipulation. TRON bridges these two paradigms. We use intrinsic decomposition priors from a learned inverse rendering model to regularize the material properties of a Gaussian field, and repurpose a ray tracer to provide radiometric guidance rather than final pixels. By treating this output as a structured 3D scaffold, we empower a lightweight neural renderer to bridge the domain gap between shading-model constrained estimates and photorealistic output. Our key insight is that the combination of explicit 3D knowledge with robust material priors provides speed and controllability, while neural rendering enables the synthesis of photorealistic images. To support real-world scenarios, we train our neural renderer with a multi-stage strategy consisting of large-scale pretraining and targeted fine-tuning on a newly constructed dataset of 2.1M rendered synthetic and real-world frames from 3D reconstructions. TRON outperforms Gaussian-based relighting methods in realism, and prior neural renderers in editability and speed. To the best of our knowledge, TRON is the first method to enable practical interactive applications in captured 3D environments, offering realistic appearance under dynamic geometric, lighting and material conditions.

Problem

• neural rendering
• 3D Gaussian
• relighting
• interactive editing
• photorealistic rendering

Innovation

• Neural Rendering
• 3D Gaussian Splatting
• Inverse Rendering
• Ray Tracing
• Relighting