Diffusion Prism: Enhancing Diversity and Morphology Consistency in Mask-to-Image Diffusion

2025-01-01

AI Summary

To address insufficient diversity and morphological distortion in low-entropy binary mask-to-image generation, this paper proposes Diffusion Prism—a training-free diffusion-based framework. Methodologically, it leverages the novel observation that minute artificial noise significantly enhances morphological consistency during denoising; establishes a mask-driven, training-free diffusion enhancement paradigm to overcome the diversity bottleneck in controllable diffusion under sparse input conditions; and integrates biologically inspired priors (e.g., nanodendritic structures) via noise modulation of pretrained diffusion models and multi-scale morphological constraints for high-fidelity synthesis. Evaluated on diverse biological patterns, Diffusion Prism achieves a 37% improvement in morphological fidelity and a 2.1× increase in visual diversity over baseline methods—including ControlNet and T2I-Adapter—demonstrating superior controllability, fidelity, and generative diversity without any fine-tuning.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation

Application Category

• Security and Privacy: Privacy-enhancing technologies
• User Modeling, Personalization and Recommendation: Accountability, Transparency, and Ethics for personalization
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

The emergence of generative AI and controllable diffusion has made image-to-image synthesis increasingly practical and efficient. However, when input images exhibit low entropy and sparse, the inherent characteristics of diffusion models often result in limited diversity. This constraint significantly interferes with data augmentation. To address this, we propose Diffusion Prism, a training-free framework that efficiently transforms binary masks into realistic and diverse samples while preserving morphological features. We explored that a small amount of artificial noise will significantly assist the image-denoising process. To prove this novel mask-to-image concept, we use nano-dendritic patterns as an example to demonstrate the merit of our method compared to existing controllable diffusion models. Furthermore, we extend the proposed framework to other biological patterns, highlighting its potential applications across various fields.

Problem

• Generative AI
• Controllable Diffusion
• Image Variety Enhancement

Innovation

• Diffusion Prism
• Shape Consistency
• Pattern Diversity