neural rendering

Existing virtual cameras lack differentiable modeling of optical properties and intrinsic parameters, leading to distorted optical artifact simulation and limited sim-to-real transfer performance. This paper introduces DiffPhysCam—the first differentiable physical camera simulator enabling gradient-based optimization—unifying forward and inverse rendering via multi-stage optical modeling, including defocus blur. Its key innovation lies in embedding tunable calibration parameters (e.g., focal length, aperture, image distance) directly into a physics-based imaging pipeline, while tightly coupling multi-physics simulation with inverse optimization algorithms. This enables digital twin scene reconstruction from real images and end-to-end optimization of 3D geometry and material properties. Evaluated on synthetic datasets and autonomous driving navigation tasks, DiffPhysCam significantly improves robotic visual perception robustness and sim-to-real consistency.

Single-image 3D scene generation with multiple objects faces severe challenges including heavy occlusion and object coupling-induced geometric distortion. To address these, we propose a two-stage differentiable framework: first, leveraging off-the-shelf image-to-3D models to independently reconstruct per-object meshes; second, jointly optimizing global scene layout via differentiable rendering, incorporating a novel optimal transport-driven long-range appearance loss and a high-level semantic loss in a synergistic constraint mechanism—enabling unified modeling of object-level geometric independence and scene-level structural consistency. Our approach integrates differentiable rendering, optimal transport theory, and semantics-guided gradient optimization. Evaluated on multi-object benchmarks, our method significantly improves geometric detail fidelity, object separation, and global coherence, outperforming state-of-the-art single-image 3D generation methods both quantitatively and qualitatively.

Existing 3D generative models neglect scene-specific constraints, resulting in synthetic assets that fail to integrate naturally into real-world environments. To address this, we propose a scene-conditioned framework for 3D object style transfer and compositing. Our method jointly optimizes object texture and environment lighting via differentiable ray tracing, while leveraging image priors from pre-trained text-to-image diffusion models (e.g., Stable Diffusion) to ensure geometric–photometric consistency and semantic adaptability in an end-to-end manner. Crucially, this work establishes the first tight coupling between 3D stylization and 2D scene semantics, enabling dynamic, context-aware relighting and restyling of a single 3D object across diverse semantic settings (e.g., summer/winter, fantasy/futuristic). We validate our approach on varied indoor/outdoor scenes and arbitrary 3D objects, demonstrating substantial improvements in visual realism, physical plausibility, and artistic controllability of composited imagery.

Existing 3D reconstruction methods suffer from low efficiency and limited quality due to indirect geometric learning and coupled geometry-appearance modeling. To address this, we propose the first end-to-end jointly optimized framework integrating explicit triangular meshes with 3D Gaussian points: differentiable 3D Gaussians are rigidly bound to mesh faces and jointly optimized under photometric supervision to reconstruct both geometry and surface appearance. Our approach breaks the conventional paradigm of decoupled geometry and appearance modeling, enabling efficient, high-fidelity reconstruction and real-time rendering. Quantitatively, it achieves a +1.8 dB PSNR improvement over prior methods on the DTU and BlendedMVS benchmarks. Moreover, the framework supports interactive mesh editing and incremental updates for dynamic scenes, significantly enhancing reconstruction efficiency and editing flexibility.

This work addresses the challenge of synthesizing high-fidelity, photorealistic images from coarse layout edits. To mitigate second-order artifacts—including illumination mismatch, missing shadows, and physically implausible object interactions—the authors propose a diffusion-based inpainting method leveraging video temporal modeling. The method introduces a novel dual-motion modeling mechanism—optical-flow-guided warping coupled with hierarchical feature injection—supervised by paired video frames, enabling joint optimization of layout alignment, illumination consistency, and physically grounded object interactions. By integrating a pre-trained diffusion model, layout-constrained fine-tuning, and a dynamically constructed video dataset, the approach achieves fine-grained detail transfer and multi-factor coherent generation. Experiments demonstrate significant improvements in output photorealism, geometric consistency, and scene plausibility, while preserving object identity and texture fidelity.

Existing video generation methods struggle to simultaneously achieve 3D consistency and visual photorealism, while conventional 3D rendering, though geometrically accurate, often lacks fine-grained realistic details. This work proposes a video diffusion-based framework that enforces structural alignment through geometric conditioning and introduces an IC-LoRA fine-tuning mechanism coupled with an anchor-frame propagation strategy to construct paired training data. This approach enables photorealistic enhancement for both anchor-free sequences and dynamic objects. Evaluated on complex GTA-V scenes, the method significantly outperforms current video editing techniques by substantially improving global material quality, lighting, and textural realism—while preserving the original geometry, motion dynamics, and identity consistency.

This work addresses the challenge of simultaneously achieving high-quality novel view synthesis and relighting from a single image while generating temporally consistent videos. To this end, it introduces the first unified video diffusion model that jointly controls viewpoint and illumination within a single generative framework by explicitly conditioning on user-specified camera trajectories and environmental lighting maps. The method concurrently outputs relit novel-view frames along with their albedo counterparts, streamlining the pipeline while preserving high fidelity. Experimental results demonstrate that the proposed approach matches or exceeds the visual quality of current state-of-the-art methods, while ensuring both spatiotemporal consistency and spatial alignment across generated video sequences.

This work addresses the challenge of efficiently and accurately reconstructing both 3D geometry and reflectance properties of objects in a unified framework. It presents the first differentiable pipeline that jointly optimizes adaptive 4D structured illumination design and geometry-appearance reconstruction in an end-to-end manner. The method integrates a unified spatial-angular structured light pattern, a single-camera setup, a histogram-based per-pixel probabilistic model, and differentiable rendering, while optimizing illumination strategies through a joint consistency constraint between physical measurements and simulations. Experiments demonstrate that the approach achieves superior depth reconstruction accuracy compared to state-of-the-art methods across a variety of complex shapes and materials, and produces reflectance parameters that closely match ground-truth appearance in real captured images.

Existing video prediction models operating in discrete pixel space often rely on mean squared error (MSE) loss, which tends to produce overly smoothed predictions with diminished detail. To address this limitation, this work proposes a novel Predictive Differentiable Rendering (PDR) paradigm that introduces 2D Gaussian differentiable rendering into video prediction for the first time. The approach balances continuous representation and discrete prediction through a lightweight, plug-and-play adapter, PredGS, coupled with an efficient CUDA-accelerated renderer, predgsplat. By jointly optimizing L1 loss and structural similarity (SSIM), the method significantly enhances visual fidelity, detail preservation, and prediction accuracy across multiple benchmarks—including TaxiBJ, WeatherBench, KTH, and Human3.6M—while achieving up to a 10× speedup in rendering compared to baseline methods.

This work addresses the challenge of maintaining structural consistency in large-scale 3D scene generation from text, a limitation inherent in current text-to-image/video methods due to their lack of explicit geometric modeling. The authors propose a geometry-first, two-stage framework: first generating an environment-level mesh scaffold—such as walls and floors—from textual descriptions, then conditioning a diffusion model on this scaffold to synthesize photorealistic images while jointly performing semantic segmentation and object reconstruction. By leveraging an explicit geometric backbone, the approach effectively decouples structure from appearance, enabling high object diversity, strong global 3D consistency, and photorealistic detail across arbitrarily scaled multi-room scenes. This facilitates the construction of navigable, realistic 3D environments directly from language prompts.