Any-to-Any Vision-Language Model for Multimodal X-ray Imaging and Radiological Report Generation

2025-05-02

AI Summary

General vision-language generative models exhibit poor adaptability to medical imaging, producing anatomically implausible outputs and clinically inconsistent semantics. Method: We propose the first end-to-end multimodal framework enabling bidirectional, arbitrary-modality generation between X-ray images and clinical reports. Our approach integrates diffusion modeling with a cross-modal alignment architecture, performs domain-adaptive training on MIMIC-CXR, and introduces anatomy-aware structural constraints and medical entity consistency optimization—operating without paired supervision. Contribution/Results: Quantitatively, our method reduces FID by 32% and improves BLEU-4 by 19%. In downstream classification of five diseases—including pneumonia and pneumothorax—it achieves 91.7% accuracy, surpassing the real-data baseline by 0.4 percentage points, thereby demonstrating the clinical validity and utility of the generated data.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Generative models have revolutionized Artificial Intelligence (AI), particularly in multimodal applications. However, adapting these models to the medical domain poses unique challenges due to the complexity of medical data and the stringent need for clinical accuracy. In this work, we introduce a framework specifically designed for multimodal medical data generation. By enabling the generation of multi-view chest X-rays and their associated clinical report, it bridges the gap between general-purpose vision-language models and the specialized requirements of healthcare. Leveraging the MIMIC-CXR dataset, the proposed framework shows superior performance in generating high-fidelity images and semantically coherent reports. Our quantitative evaluation reveals significant results in terms of FID and BLEU scores, showcasing the quality of the generated data. Notably, our framework achieves comparable or even superior performance compared to real data on downstream disease classification tasks, underlining its potential as a tool for medical research and diagnostics. This study highlights the importance of domain-specific adaptations in enhancing the relevance and utility of generative models for clinical applications, paving the way for future advancements in synthetic multimodal medical data generation.

Problem

• Adapting generative models for complex medical data accuracy
• Bridging vision-language models with healthcare-specific multimodal needs
• Enhancing clinical relevance of synthetic X-rays and reports

Innovation

• Any-to-any vision-language model for medical data
• Generates multi-view X-rays and clinical reports
• Leverages MIMIC-CXR for high-fidelity outputs