NoiseShift: Resolution-Aware Noise Recalibration for Better Low-Resolution Image Generation

2025-10-02

AI Summary

Text-to-image diffusion models suffer significant performance degradation at low resolutions, primarily due to perceptual shifts in the noise scheduler: identical noise levels remove more semantic signal at lower resolutions, causing train-test distribution mismatch. To address this, we propose NoiseShift—a training-free, model- and sampler-agnostic method that performs resolution-aware noise recalibration. By dynamically adjusting noise levels according to input resolution, NoiseShift ensures consistent semantic preservation across scales. The approach is fully compatible with Stable Diffusion and Flux-family models. On LAION-COCO, it improves FID by 15.89% for SD3.5 and 8.56% for SD3; on CelebA, SD3.5 achieves a 10.36% FID reduction. These gains demonstrate substantially enhanced cross-resolution generalization. NoiseShift provides a plug-and-play, lightweight solution for efficient multi-resolution image generation without architectural or training modifications.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Text-to-image diffusion models trained on a fixed set of resolutions often fail to generalize, even when asked to generate images at lower resolutions than those seen during training. High-resolution text-to-image generators are currently unable to easily offer an out-of-the-box budget-efficient alternative to their users who might not need high-resolution images. We identify a key technical insight in diffusion models that when addressed can help tackle this limitation: Noise schedulers have unequal perceptual effects across resolutions. The same level of noise removes disproportionately more signal from lower-resolution images than from high-resolution images, leading to a train-test mismatch. We propose NoiseShift, a training-free method that recalibrates the noise level of the denoiser conditioned on resolution size. NoiseShift requires no changes to model architecture or sampling schedule and is compatible with existing models. When applied to Stable Diffusion 3, Stable Diffusion 3.5, and Flux-Dev, quality at low resolutions is significantly improved. On LAION-COCO, NoiseShift improves SD3.5 by 15.89%, SD3 by 8.56%, and Flux-Dev by 2.44% in FID on average. On CelebA, NoiseShift improves SD3.5 by 10.36%, SD3 by 5.19%, and Flux-Dev by 3.02% in FID on average. These results demonstrate the effectiveness of NoiseShift in mitigating resolution-dependent artifacts and enhancing the quality of low-resolution image generation.

Problem

• Addresses resolution mismatch in diffusion model noise schedulers
• Improves low-resolution image generation without architectural changes
• Mitigates resolution-dependent artifacts in text-to-image models

Innovation

• Recalibrates noise levels based on resolution size
• Requires no model architecture or schedule changes
• Improves low-resolution image quality across diffusion models