PARROT: An Open Multilingual Radiology Reports Dataset

2025-07-25

AI Summary

Current radiology NLP research is hindered by the scarcity of open, cross-lingual, geographically diverse, and regulatory-compliant datasets. To address this, we introduce PARROT—the largest publicly available synthetic multilingual radiology report dataset to date—comprising 2,658 structured reports authored by 76 radiologists across 21 countries and 13 languages, annotated with ICD-10 codes, anatomical regions, and imaging modalities, and accompanied by English translations. PARROT overcomes privacy constraints while enabling robust multilingual AI development and equitable evaluation. Its clinical fidelity was validated via double-blind human–machine discrimination experiments: non-expert annotators achieved 53.9% accuracy and radiologists 56.9%, near chance level, confirming high realism. PARROT fills a critical gap in multicenter, multilingual, clinically grounded radiology text resources, establishing foundational infrastructure for global radiology NLP research.

Technology Category

• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Rationale and Objectives: To develop and validate PARROT (Polyglottal Annotated Radiology Reports for Open Testing), a large, multicentric, open-access dataset of fictional radiology reports spanning multiple languages for testing natural language processing applications in radiology. Materials and Methods: From May to September 2024, radiologists were invited to contribute fictional radiology reports following their standard reporting practices. Contributors provided at least 20 reports with associated metadata including anatomical region, imaging modality, clinical context, and for non-English reports, English translations. All reports were assigned ICD-10 codes. A human vs. AI report differentiation study was conducted with 154 participants (radiologists, healthcare professionals, and non-healthcare professionals) assessing whether reports were human-authored or AI-generated. Results: The dataset comprises 2,658 radiology reports from 76 authors across 21 countries and 13 languages. Reports cover multiple imaging modalities (CT: 36.1%, MRI: 22.8%, radiography: 19.0%, ultrasound: 16.8%) and anatomical regions, with chest (19.9%), abdomen (18.6%), head (17.3%), and pelvis (14.1%) being most prevalent. In the differentiation study, participants achieved 53.9% accuracy (95% CI: 50.7%-57.1%) in distinguishing between human and AI-generated reports, with radiologists performing significantly better (56.9%, 95% CI: 53.3%-60.6%, p<0.05) than other groups. Conclusion: PARROT represents the largest open multilingual radiology report dataset, enabling development and validation of natural language processing applications across linguistic, geographic, and clinical boundaries without privacy constraints.

Problem

• Creating multilingual radiology reports dataset for NLP testing
• Validating human vs AI-generated report differentiation accuracy
• Enabling NLP development across languages without privacy issues

Innovation

• Multilingual fictional radiology reports dataset
• Human vs AI report differentiation study
• Open-access with ICD-10 coded metadata