Image Super-Resolution with Text Prompt Diffusion

2023-11-24

arXiv.org

AI Summary

Image super-resolution (SR) under unknown degradations is fundamentally limited by the difficulty of accurately estimating degradation characteristics from low-resolution inputs. To address this, we propose a novel text-guided SR paradigm that explicitly incorporates discretized textual degradation descriptions—derived via binning—as a structured prior into the reconstruction process, thereby alleviating the degradation modeling bottleneck. We introduce the first text–image paired SR benchmark and present PromptSR, a unified framework integrating pretrained language models (T5/CLIP), text-to-image generation principles, differentiable degradation modeling, and a diffusion-based reconstruction architecture for language-driven, controllable SR. Extensive experiments demonstrate that PromptSR significantly outperforms state-of-the-art unsupervised and degradation-aware methods on both synthetic and real-world images, achieving substantial gains in PSNR and SSIM. The code and dataset are publicly released.

Abstract

Image super-resolution (SR) methods typically model degradation to improve reconstruction accuracy in complex and unknown degradation scenarios. However, extracting degradation information from low-resolution images is challenging, which limits the model performance. To boost image SR performance, one feasible approach is to introduce additional priors. Inspired by advancements in multi-modal methods and text prompt image processing, we introduce text prompts to image SR to provide degradation priors. Specifically, we first design a text-image generation pipeline to integrate text into the SR dataset through the text degradation representation and degradation model. The text representation applies a discretization manner based on the binning method to describe the degradation abstractly. This method maintains the flexibility of the text and is user-friendly. Meanwhile, we propose the PromptSR to realize the text prompt SR. The PromptSR utilizes the pre-trained language model (e.g., T5 or CLIP) to enhance restoration. We train the PromptSR on the generated text-image dataset. Extensive experiments indicate that introducing text prompts into SR, yields excellent results on both synthetic and real-world images. Code is available at: https://github.com/zhengchen1999/PromptSR.

Problem

• Enhance image super-resolution using text prompts
• Integrate text degradation priors into SR models
• Improve SR accuracy with multi-modal language models

Innovation

• Text prompts enhance image super-resolution accuracy.
• Multi-modal large language models generate SR prompts.
• Pre-trained language models improve text comprehension.