Noise-Free One-Step LoRA for Task-Driven Image Restoration with Diffusion Priors

2026-07-28

AI Summary

This work addresses the challenge that image degradation not only impairs visual quality but also degrades performance in downstream high-level vision tasks. Existing diffusion-based restoration methods often rely on stochastic noise during inference, compromising task consistency. To overcome this limitation, the authors propose a deterministic, one-step forward inference framework that eliminates the need for injected noise by integrating a pre-trained diffusion prior with a lightweight LoRA adaptation module. They further introduce a task-preserving GAN training strategy. This approach represents the first task-driven deterministic diffusion-based restoration method and demonstrates that LoRA outperforms ControlNet-style conditional mechanisms in preserving task-relevant semantics. Experiments show consistent superiority over existing task-driven image restoration (TDIR) methods across image classification, segmentation, and detection, with strong generalization on real-world degraded images and OCR tasks.

Abstract

Degraded images not only reduce visual quality but also impair downstream high-level vision tasks. Task-driven image restoration (TDIR) addresses this issue by jointly optimizing restoration quality and task performance. Recent works show that pretrained diffusion priors benefit TDIR, yet diffusion-based restoration is inherently stochastic, as the sampling process depends on a random noise term, which can undermine task consistency. In this paper, we show that a deterministic, noise-free one-step forward pass with pretrained diffusion priors can substantially improve TDIR, but the benefit critically depends on the adaptation module: LoRA yields consistent gains, whereas ControlNet-style conditioning does not. This enables one-step forwarding that surpasses conventional multi-step diffusion TDIR baselines. Furthermore, we introduce a task-preserving GAN training strategy that improves perceptual quality without sacrificing task performance. Extensive experiments on classification, segmentation, and detection demonstrate consistent gains over prior TDIR methods, and we further validate generalization on real-world degraded images and OCR.

Problem

• task-driven image restoration
• diffusion priors
• stochastic sampling
• task consistency
• image degradation

Innovation

• Noise-Free One-Step
• LoRA
• Task-Driven Image Restoration
• Diffusion Priors
• Task-Preserving GAN