Towards Physically-Based Sky-Modeling For Image Based Lighting

2025-12-15

AI Summary

Existing sky models struggle to simultaneously achieve photorealism and full dynamic range (FDR, 22 f-stops) physical lighting fidelity—particularly in image-based lighting (IBL) relighting, where tone, shadow, and highlight consistency are severely compromised. This paper introduces AllSky, the first all-weather, full-FDR sky model trained directly from physically captured HDR imagery. We innovatively embed physics-based lighting constraints into a deep neural network architecture, incorporating multimodal input encoding (solar parameters + cloud maps), differentiable tone mapping, and lighting-aware loss functions. Furthermore, we establish a dedicated IBL-oriented evaluation paradigm. Experiments demonstrate that AllSky achieves close visual and radiometric fidelity to real-world HDR captures in relighting tasks: it reduces relighting error by 47% under FDR conditions and sets a new state-of-the-art.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Accurate environment maps are a key component for rendering photorealistic outdoor scenes with coherent illumination. They enable captivating visual arts, immersive virtual reality, and a wide range of engineering and scientific applications. Recent works have extended sky-models to be more comprehensive and inclusive of cloud formations but, as we demonstrate, existing methods fall short in faithfully recreating natural skies. Though in recent years the visual quality of DNN-generated High Dynamic Range Imagery (HDRI) has greatly improved, the environment maps generated by DNN sky-models do not re-light scenes with the same tones, shadows, and illumination as physically captured HDR imagery. In this work, we demonstrate progress in HDR literature to be tangential to sky-modelling as current works cannot support both photorealism and the 22 f-stops required for the Full Dynamic Range (FDR) of outdoor illumination. We achieve this by proposing AllSky, a flexible all-weather sky-model learned directly from physically captured HDRI which we leverage to study the input modalities, tonemapping, conditioning, and evaluation of sky-models. Per user-controlled positioning of the sun and cloud formations, AllSky expands on current functionality by allowing for intuitive user control over environment maps and achieves state-of-the-art sky-model performance. Through our proposed evaluation, we demonstrate existing DNN sky-models are not interchangeable with physically captured HDRI or parametric sky-models, with current limitations being prohibitive of scalability and accurate illumination in downstream applications

Problem

• Develops a physically-based sky-model for realistic outdoor lighting
• Addresses limitations of DNN-generated HDR imagery in accurate scene relighting
• Enables user-controlled environment maps with full dynamic range fidelity

Innovation

• Learned from physically captured HDRI for accurate sky-modeling
• Allows intuitive user control over sun and cloud positioning
• Achieves state-of-the-art performance with full dynamic range support