Technology Category
Computer Vision: Diffusion Models for VisionMachine Learning: Learning on the Edge & Model CompressionSearch and Optimization: Learning to Search
NanoSD: Edge Efficient Foundation Model for Real Time Image Restoration
🤖 AI Summary
This work addresses the degradation of generalization and the inability to achieve real-time edge inference in existing lightweight diffusion models, which often stems from aggressive U-Net compression or reduced diffusion steps that disrupt the latent manifold structure. To overcome these limitations, the authors propose a holistic pipeline co-compression strategy that synergistically optimizes the diffusion process through network surgery, feature-level generative distillation, and structured scaling of both the U-Net and VAE components. The study reveals that parameter count alone does not dictate hardware efficiency; instead, architectural balance, effective feature routing, and preservation of the latent space are critical. This approach yields the first family of multi-task, real-time lightweight diffusion models (130M–315M parameters), achieving as low as 20ms inference latency on mobile NPUs and setting new state-of-the-art results across diverse tasks—including super-resolution, deblurring, face restoration, and monocular depth estimation—while maintaining high perceptual quality and deployment efficiency as a plug-and-play backbone.
Technology Category
Application Category
📝 Abstract
Latent diffusion models such as Stable Diffusion 1.5 offer strong generative priors that are highly valuable for image restoration, yet their full pipelines remain too computationally heavy for deployment on edge devices. Existing lightweight variants predominantly compress the denoising U-Net or reduce the diffusion trajectory, which disrupts the underlying latent manifold and limits generalization beyond a single task. We introduce NanoSD, a family of Pareto-optimal diffusion foundation models distilled from Stable Diffusion 1.5 through network surgery, feature-wise generative distillation, and structured architectural scaling jointly applied to the U-Net and the VAE encoder-decoder. This full-pipeline co-design preserves the generative prior while producing models that occupy distinct operating points along the accuracy-latency-size frontier (e.g., 130M-315M parameters, achieving real-time inference down to 20ms on mobile-class NPUs). We show that parameter reduction alone does not correlate with hardware efficiency, and we provide an analysis revealing how architectural balance, feature routing, and latent-space preservation jointly shape true on-device latency. When used as a drop-in backbone, NanoSD enables state-of-the-art performance across image super-resolution, image deblurring, face restoration, and monocular depth estimation, outperforming prior lightweight diffusion models in both perceptual quality and practical deployability. NanoSD establishes a general-purpose diffusion foundation model family suitable for real-time visual generation and restoration on edge devices.
Problem
Research questions and friction points this paper is trying to address.
edge devices
real-time image restoration
latent diffusion models
model efficiency
generative priors
Innovation
Methods, ideas, or system contributions that make the work stand out.
diffusion foundation model
edge-efficient AI
generative distillation
full-pipeline co-design
real-time image restoration
🔎 Similar Papers
No similar papers found.
💼 Related Jobs
