Manifold-Constrained Noise Optimization for Diverse Diffusion Sampling

2026-07-26

AI Summary

This work addresses the limited diversity in few-step distilled diffusion models under a single prompt, which often arises from initial noise optimization that neglects geometric priors and frequency sensitivity. The authors propose MoNO, a novel approach that constrains the initial noise to a low-dimensional, high-quality manifold—specifically, a low-frequency affine sphere—and performs large-step updates via Riemannian geodesics. This strategy simultaneously enhances generation diversity and fidelity without requiring additional quality-control objectives. By integrating manifold constraints, frequency-aware noise optimization, and guidance from complementary visual features, MoNO significantly improves prompt-level diversity across multiple distilled text-to-image models while preserving image quality and achieving substantially faster convergence than existing noise optimization methods.

Abstract

Few-step distilled diffusion models generate high-quality images quickly, but often lose per-prompt diversity, producing near-identical samples across random seeds. Optimizing the initial noise at inference time offers an appealing way to recover this diversity, yet existing methods directly update the initial noise in an unconstrained Euclidean space, ignoring both the geometry of the Gaussian prior and the model's sensitivity to noise frequencies. They therefore introduce auxiliary quality-control objectives to maintain generation fidelity, adding compute and weighting hyperparameters while still requiring conservative updates to prevent degradation. In this work, we propose MoNO, a training-free method that performs Manifold-constrained Noise Optimization on a low-dimensional, quality-stabilizing noise manifold. MoNO sequentially optimizes each new initial noise so that its predicted visual feature complements previous generations, while Riemannian updates on an affine low-frequency sphere preserve prior likelihood and fix unstable high-frequency components by construction. This enables large geodesic steps, removes the need for auxiliary quality-control objectives, and converges in far fewer iterations than prior noise-optimization methods. Experiments with multiple distilled text-to-image diffusion models show that MoNO consistently improves per-prompt diversity while maintaining image quality.

Problem

• diversity
• diffusion models
• noise optimization
• few-step generation
• image generation

Innovation

• manifold-constrained optimization
• diffusion sampling
• noise optimization
• Riemannian geometry
• per-prompt diversity