See through the Dark: Learning Illumination-affined Representations for Nighttime Occupancy Prediction

2025-05-27

AI Summary

To address the severe performance degradation of 3D occupancy prediction under low-light nighttime conditions, this paper proposes LIAR, the first framework introducing a lighting-adaptive representation learning paradigm. Methodologically, LIAR integrates three novel components: Selective Low-Light Image Enhancement (SLLIE), Lighting-Guided 2D Sampling (2D-IGS), and Lighting-Driven 3D Projection (3D-IDP). These modules jointly incorporate lighting prior modeling, dynamic feature sampling offsets, construction of a lighting intensity field in the bird’s-eye view (BEV) space, and residual query enhancement—implemented via a multi-scale visual encoder and Transformer architecture for robust perception. Evaluated on both real-world and synthetic nighttime datasets, LIAR achieves state-of-the-art performance in both semantic accuracy and geometric completeness, effectively mitigating feature degradation caused by underexposure and overexposure.

Technology Category

• Computer Vision: 3D Computer Vision
• Search and Optimization: Learning to Search
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Occupancy prediction aims to estimate the 3D spatial distribution of occupied regions along with their corresponding semantic labels. Existing vision-based methods perform well on daytime benchmarks but struggle in nighttime scenarios due to limited visibility and challenging lighting conditions. To address these challenges, we propose extbf{LIAR}, a novel framework that learns illumination-affined representations. LIAR first introduces Selective Low-light Image Enhancement (SLLIE), which leverages the illumination priors from daytime scenes to adaptively determine whether a nighttime image is genuinely dark or sufficiently well-lit, enabling more targeted global enhancement. Building on the illumination maps generated by SLLIE, LIAR further incorporates two illumination-aware components: 2D Illumination-guided Sampling (2D-IGS) and 3D Illumination-driven Projection (3D-IDP), to respectively tackle local underexposure and overexposure. Specifically, 2D-IGS modulates feature sampling positions according to illumination maps, assigning larger offsets to darker regions and smaller ones to brighter regions, thereby alleviating feature degradation in underexposed areas. Subsequently, 3D-IDP enhances semantic understanding in overexposed regions by constructing illumination intensity fields and supplying refined residual queries to the BEV context refinement process. Extensive experiments on both real and synthetic datasets demonstrate the superior performance of LIAR under challenging nighttime scenarios. The source code and pretrained models are available href{https://github.com/yanzq95/LIAR}{here}.

Problem

• Predicting 3D occupancy in low-visibility nighttime conditions
• Addressing feature degradation in underexposed dark regions
• Improving semantic understanding in overexposed bright areas

Innovation

• Selective Low-light Image Enhancement (SLLIE)
• 2D Illumination-guided Sampling (2D-IGS)
• 3D Illumination-driven Projection (3D-IDP)