Prompt-Guided Latent Diffusion with Predictive Class Conditioning for 3D Prostate MRI Generation

2025-06-11

AI Summary

To address the limitations of latent diffusion models (LDMs) in medical image generation—namely, their reliance on generic text encoders, transfer from non-medical pre-trained models, and dependence on large-scale annotated datasets—this paper proposes a few-shot, pathology-conditioned 3D prostate MRI synthesis method. We introduce a novel dual-path category conditioning mechanism integrating textual semantics and pathology labels, a lightweight large language model adapter (CCELLA), and a data-efficient joint loss training framework. Evaluated under extreme data scarcity, our model achieves a 3D Fréchet Inception Distance (FID) of 0.025—significantly outperforming the baseline (0.071). Synthesized images boost downstream cancer classification accuracy to 74%, and models trained exclusively on synthetic data match the performance of those trained on real data. This advances clinical applicability and scientific reproducibility of medical generative models.

Technology Category

Machine Learning: Large Multimodal Models (LMMs)Computer Vision: Diffusion Models for VisionNatural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMsSearch and Retrieval-Augmented AI: Large language models for searchUser Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Latent diffusion models (LDM) could alleviate data scarcity challenges affecting machine learning development for medical imaging. However, medical LDM training typically relies on performance- or scientific accessibility-limiting strategies including a reliance on short-prompt text encoders, the reuse of non-medical LDMs, or a requirement for fine-tuning with large data volumes. We propose a Class-Conditioned Efficient Large Language model Adapter (CCELLA) to address these limitations. CCELLA is a novel dual-head conditioning approach that simultaneously conditions the LDM U-Net with non-medical large language model-encoded text features through cross-attention and with pathology classification through the timestep embedding. We also propose a joint loss function and a data-efficient LDM training framework. In combination, these strategies enable pathology-conditioned LDM training for high-quality medical image synthesis given limited data volume and human data annotation, improving LDM performance and scientific accessibility. Our method achieves a 3D FID score of 0.025 on a size-limited prostate MRI dataset, significantly outperforming a recent foundation model with FID 0.071. When training a classifier for prostate cancer prediction, adding synthetic images generated by our method to the training dataset improves classifier accuracy from 69% to 74%. Training a classifier solely on our method's synthetic images achieved comparable performance to training on real images alone.

Problem

• Addresses data scarcity in medical imaging with latent diffusion models
• Improves LDM performance with limited data and annotations
• Enhances pathology-conditioned medical image synthesis quality

Innovation

• Dual-head conditioning with cross-attention and timestep embedding
• Joint loss function for efficient LDM training
• Data-efficient framework for limited medical datasets