A Luminance-Aware Multi-Scale Network for Polarization Image Fusion with a Multi-Scene Dataset

2025-10-28

AI Summary

To address contrast imbalance caused by luminance disparity in polarization image fusion under complex illumination, this paper proposes a luminance-aware multi-scale fusion network. Methodologically, it designs a luminance-branch-guided spatial weight matrix to achieve dynamic luminance alignment between S0 and DOLP images; introduces a global-local collaborative feature fusion mechanism; and incorporates a windowed self-attention module coupled with a luminance enhancement module to perform nonlinear correction of luminance distribution and texture features. Contributions include: (1) releasing MSP—the first large-scale polarization image fusion dataset comprising one thousand image pairs across 17 scene categories; and (2) achieving state-of-the-art performance on MSP, PIF, and GAND benchmarks, with average improvements of 8.57%–63.53% in MS-SSIM and standard deviation (SD) metrics, validated by both quantitative and qualitative evaluations.

Technology Category

• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Security and Privacy: Large-scale security measurements

Abstract

Polarization image fusion combines S0 and DOLP images to reveal surface roughness and material properties through complementary texture features, which has important applications in camouflage recognition, tissue pathology analysis, surface defect detection and other fields. To intergrate coL-Splementary information from different polarized images in complex luminance environment, we propose a luminance-aware multi-scale network (MLSN). In the encoder stage, we propose a multi-scale spatial weight matrix through a brightness-branch , which dynamically weighted inject the luminance into the feature maps, solving the problem of inherent contrast difference in polarized images. The global-local feature fusion mechanism is designed at the bottleneck layer to perform windowed self-attention computation, to balance the global context and local details through residual linking in the feature dimension restructuring stage. In the decoder stage, to further improve the adaptability to complex lighting, we propose a Brightness-Enhancement module, establishing the mapping relationship between luminance distribution and texture features, realizing the nonlinear luminance correction of the fusion result. We also present MSP, an 1000 pairs of polarized images that covers 17 types of indoor and outdoor complex lighting scenes. MSP provides four-direction polarization raw maps, solving the scarcity of high-quality datasets in polarization image fusion. Extensive experiment on MSP, PIF and GAND datasets verify that the proposed MLSN outperms the state-of-the-art methods in subjective and objective evaluations, and the MS-SSIM and SD metircs are higher than the average values of other methods by 8.57%, 60.64%, 10.26%, 63.53%, 22.21%, and 54.31%, respectively. The source code and dataset is avalable at https://github.com/1hzf/MLS-UNet.

Problem

• Fusing polarization images in complex luminance environments for enhanced feature extraction
• Addressing inherent contrast differences in polarized images through dynamic luminance injection
• Solving dataset scarcity with comprehensive multi-scene polarization image collection

Innovation

• Luminance-aware multi-scale network for polarization image fusion
• Multi-scale spatial weight matrix dynamically injects luminance
• Brightness-Enhancement module establishes luminance-texture mapping relationship