EigenSR: Eigenimage-Bridged Pre-Trained RGB Learners for Single Hyperspectral Image Super-Resolution

2024-09-06

arXiv.org

AI Summary

To address the bottleneck in hyperspectral image (HSI) super-resolution caused by scarce labeled data, this paper proposes a two-stage decoupled framework leveraging transfer learning from RGB pre-trained models. First, low-resolution HSI is mapped to a compact spatial-spectral representation domain via eigenimage decomposition, where an off-the-shelf RGB super-resolution model is fine-tuned. Second, iterative spectral regularization (ISR) and a spectral low-rank constraint are introduced to explicitly enforce spectral fidelity. To our knowledge, this is the first work to successfully transfer priors from large-scale RGB foundation models to HSI reconstruction, achieving simultaneous spatial detail enhancement and strict spectral consistency. Extensive experiments on multiple benchmark datasets demonstrate state-of-the-art performance, with significant improvements in PSNR, SSIM, SAM, and ERGAS. The method further exhibits strong generalization capability and higher inference efficiency.

Technology Category

• Computer Vision: Representation Learning for Vision
• Machine Learning: Semi-Supervised Learning
• Knowledge Representation and Reasoning: Applications

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Single hyperspectral image super-resolution (single-HSI-SR) aims to improve the resolution of a single input low-resolution HSI. Due to the bottleneck of data scarcity, the development of single-HSI-SR lags far behind that of RGB natural images. In recent years, research on RGB SR has shown that models pre-trained on large-scale benchmark datasets can greatly improve performance on unseen data, which may stand as a remedy for HSI. But how can we transfer the pre-trained RGB model to HSI, to overcome the data-scarcity bottleneck? Because of the significant difference in the channels between the pre-trained RGB model and the HSI, the model cannot focus on the correlation along the spectral dimension, thus limiting its ability to utilize on HSI. Inspired by the HSI spatial-spectral decoupling, we propose a new framework that first fine-tunes the pre-trained model with the spatial components (known as eigenimages), and then infers on unseen HSI using an iterative spectral regularization (ISR) to maintain the spectral correlation. The advantages of our method lie in: 1) we effectively inject the spatial texture processing capabilities of the pre-trained RGB model into HSI while keeping spectral fidelity, 2) learning in the spectral-decorrelated domain can improve the generalizability to spectral-agnostic data, and 3) our inference in the eigenimage domain naturally exploits the spectral low-rank property of HSI, thereby reducing the complexity. This work bridges the gap between pre-trained RGB models and HSI via eigenimages, addressing the issue of limited HSI training data, hence the name EigenSR. Extensive experiments show that EigenSR outperforms the state-of-the-art (SOTA) methods in both spatial and spectral metrics.

Problem

• High Spectral Image Super-Resolution
• Pre-trained RGB Models
• Data Scarcity

Innovation

• EigenSR
• High-Spectral-Image-Super-Resolution
• Pre-trained-Model-Adaptation