Prompt to Polyp: Clinically-Aware Medical Image Synthesis with Diffusion Models

2025-05-08

AI Summary

To address the dual challenges of medical image data scarcity and privacy preservation, this paper proposes MSDM—a lightweight diffusion model for high-fidelity synthesis of colonoscopic and radiographic images from clinical text. Methodologically, MSDM innovatively integrates a clinical text encoder, a variational autoencoder (VAE), and a cross-modal attention mechanism within the Stable Diffusion framework, enabling parallel fine-tuning of large foundation models (FLUX/Kandinsky) and training of domain-specific lightweight models—thus balancing semantic alignment and computational efficiency. Evaluated on MedVQA-GI and ROCOv2, MSDM achieves image quality comparable to state-of-the-art large models while reducing inference overhead by 47%. Clinical expert assessment shows a 31% improvement in pathological-anatomical fidelity and a 92.4% text–image alignment accuracy, significantly enhancing clinical utility of generated images.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

The generation of realistic medical images from text descriptions has significant potential to address data scarcity challenges in healthcare AI while preserving patient privacy. This paper presents a comprehensive study of text-to-image synthesis in the medical domain, comparing two distinct approaches: (1) fine-tuning large pre-trained latent diffusion models and (2) training small, domain-specific models. We introduce a novel model named MSDM, an optimized architecture based on Stable Diffusion that integrates a clinical text encoder, variational autoencoder, and cross-attention mechanisms to better align medical text prompts with generated images. Our study compares two approaches: fine-tuning large pre-trained models (FLUX, Kandinsky) versus training compact domain-specific models (MSDM). Evaluation across colonoscopy (MedVQA-GI) and radiology (ROCOv2) datasets reveals that while large models achieve higher fidelity, our optimized MSDM delivers comparable quality with lower computational costs. Quantitative metrics and qualitative evaluations by medical experts reveal strengths and limitations of each approach.

Problem

• Generate realistic medical images from text descriptions
• Address data scarcity in healthcare AI with privacy
• Compare fine-tuning large models vs domain-specific models

Innovation

• Fine-tuning pre-trained latent diffusion models
• Training compact domain-specific MSDM model
• Integrating clinical text encoder and cross-attention