A Minimalist Method for Fine-tuning Text-to-Image Diffusion Models

2025-05-23

AI Summary

Fine-tuning text-to-image diffusion models often relies on trajectory caching, differentiable reward models, or complex guidance mechanisms—introducing redundancy and instability. Method: This paper proposes a minimalist paradigm that optimizes only the initial noise distribution conditioned on prompts while keeping the pre-trained diffusion model entirely frozen. It introduces the first empirical validation of the “golden noise” hypothesis and establishes a gradient-free reinforcement learning framework based on Proximal Policy Optimization (PPO), integrating parametric noise-space modeling and a prompt-conditioned noise generator—eliminating trajectory storage, reward backpropagation, and external guidance. Contribution/Results: Experiments demonstrate significant improvements in text–image alignment and image quality under low sampling steps; gains diminish gradually with increasing steps but remain stable, thereby explicitly characterizing the effective boundary of noise-space optimization.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Search and Optimization: Learning to Search
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Recent work uses reinforcement learning (RL) to fine-tune text-to-image diffusion models, improving text-image alignment and sample quality. However, existing approaches introduce unnecessary complexity: they cache the full sampling trajectory, depend on differentiable reward models or large preference datasets, or require specialized guidance techniques. Motivated by the"golden noise"hypothesis -- that certain initial noise samples can consistently yield superior alignment -- we introduce Noise PPO, a minimalist RL algorithm that leaves the pre-trained diffusion model entirely frozen and learns a prompt-conditioned initial noise generator. Our approach requires no trajectory storage, reward backpropagation, or complex guidance tricks. Extensive experiments show that optimizing the initial noise distribution consistently improves alignment and sample quality over the original model, with the most significant gains at low inference steps. As the number of inference steps increases, the benefit of noise optimization diminishes but remains present. These findings clarify the scope and limitations of the golden noise hypothesis and reinforce the practical value of minimalist RL fine-tuning for diffusion models.

Problem

• Improving text-image alignment in diffusion models
• Reducing complexity in reinforcement learning fine-tuning
• Optimizing initial noise for better sample quality

Innovation

• Uses Noise PPO for minimalist RL fine-tuning
• Learns prompt-conditioned initial noise generator
• Requires no trajectory storage or reward backpropagation