GIT-CXR: End-to-End Transformer for Chest X-Ray Report Generation

2025-01-05

AI Summary

To address the heavy reporting burden on radiologists and insufficient clinical fact accuracy in automated report generation, this paper proposes an end-to-end multimodal Transformer model incorporating curriculum learning for chest X-ray–to–diagnostic-report generation. We introduce curriculum learning—novelly applied to medical imaging natural language generation (NLG)—to progressively enhance the model’s understanding and generation of critical clinical concepts. Evaluated on the MIMIC-CXR-JPG dataset, our method achieves state-of-the-art performance across standard NLG metrics (BLEU-4, ROUGE-L, METEOR) and clinically oriented metrics (F1-macro, F1-micro, and examples-averaged F1). Experimental results demonstrate substantial improvements not only in report consistency and accessibility but, more importantly, in clinical fact accuracy. This framework establishes a clinically viable paradigm for AI-assisted radiological diagnosis.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Medical imaging is crucial for diagnosing, monitoring, and treating medical conditions. The medical reports of radiology images are the primary medium through which medical professionals attest their findings, but their writing is time consuming and requires specialized clinical expertise. The automated generation of radiography reports has thus the potential to improve and standardize patient care and significantly reduce clinicians workload. Through our work, we have designed and evaluated an end-to-end transformer-based method to generate accurate and factually complete radiology reports for X-ray images. Additionally, we are the first to introduce curriculum learning for end-to-end transformers in medical imaging and demonstrate its impact in obtaining improved performance. The experiments have been conducted using the MIMIC-CXR-JPG database, the largest available chest X-ray dataset. The results obtained are comparable with the current state-of-the-art on the natural language generation (NLG) metrics BLEU and ROUGE-L, while setting new state-of-the-art results on F1 examples-averaged, F1-macro and F1-micro metrics for clinical accuracy and on the METEOR metric widely used for NLG.

Problem

• Automated Radiology Report Generation
• Chest X-ray Processing
• Medical Time-saving

Innovation

• Artificial Intelligence
• Curriculum Learning
• Radiology Report Generation