SCALMU: Synthetically-trained Coupling of Adaptive Learned Multiplicative Updates for Hyperspectral-Multispectral Fusion

2026-05-29

AI Summary

This work addresses the challenges of hyperspectral and multispectral image fusion, particularly the scarcity of real labeled data and the limited performance of conventional methods. The authors propose SCALMU, a novel approach that, for the first time, integrates an adaptive learnable multiplicative update mechanism into an unrolled neural network framework based on coupled non-negative matrix factorization (CNMF). This design enables end-to-end optimization while preserving non-negativity and physical interpretability. Trained exclusively on synthetic data generated via the dead leaves model, SCALMU significantly outperforms existing state-of-the-art methods across multiple benchmark datasets, demonstrating superior fusion accuracy and enhanced capability in recovering fine spatial and spectral details.

Abstract

HyperSpectral-MultiSpectral Image (HSI-MSI) fusion enables high-resolution hyperspectral imaging by combining the rich spectral information of low-spatial-resolution hyperspectral images with the detailed spatial structure of multispectral images. Classical methods such as Coupled Nonnegative Matrix Factorization (CNMF) benefit from a strong physical interpretability but suffer from inferior results compared to their deep-learning counterparts. To address this limitation, we propose SCALMU (Synthetically-trained Coupling of Adaptive Learned Multiplicative Updates), a novel unrolled neural network architecture that integrates adaptive learnable matrices within the classical framework of CNMF multiplicative updates, improving its results. Due to its architectural proximity with CNMF, the resulting algorithm preserves physical interpretability and nonnegativity constraints. To overcome data scarcity for training, we additionally generate a synthetic HSI-MSI dataset via the dead leaves model, enabling synthetic supervision. SCALMU is then trained end-to-end on this dataset. Experiments demonstrate SCALMU's superiority over state-of-the-art methods on several datasets. The code is available at https://github.com/xinxinxu99/SCALMU.git

Problem

• Hyperspectral-Multispectral Fusion
• Image Fusion
• HSI-MSI
• Spatial-Spectral Resolution
• Remote Sensing

Innovation

• unrolled neural network
• adaptive learned multiplicative updates
• hyperspectral-multispectral fusion
• synthetic data generation
• nonnegative matrix factorization