HarMoE: Multi-Source Chest Radiograph Pretraining with Dataset-Disentangled Experts

2026-08-03

AI Summary

This work addresses the limitations of existing chest X-ray vision-language models, which predominantly rely on a single data source (e.g., MIMIC-CXR) and thus fail to leverage the richer, more explicit disease supervision signals available in multi-label classification datasets, often conflating clinical semantics with dataset-specific artifacts. To overcome this, the authors propose HarMoE, a novel framework that employs a dataset-aware mixture-of-experts (MoE) mechanism within a deep decoder to disentangle shared medical semantics from dataset-specific features. HarMoE further introduces masked multi-dataset supervised training based on a unified disease vocabulary. This approach enables, for the first time, effective knowledge fusion across heterogeneous chest X-ray datasets, significantly outperforming strong baselines in zero-shot classification, out-of-distribution generalization, and lesion localization tasks.

Abstract

Recent vision-language models for chest X-ray understanding are largely built on image-report alignment and therefore rely heavily on MIMIC-CXR as the dominant pretraining source. While effective at scale, this paradigm underexplores an important alternative source of supervision: a range of existing multi-label classification datasets, which provide cleaner and more explicit disease signals than free-text reports, and can offer broader pathology coverage when combined across sources. However, learning from such heterogeneous datasets is nontrivial, as differences in label ontologies, annotation protocols, acquisition pipelines, and report styles can cause models to entangle clinical semantics with dataset identity, leading to poor transfer despite increased scale. In this work, we revisit radiology VLM construction from the perspective of harmonized multi-source learning. We propose HarMoE, a dataset-aware mixture-of-experts framework that learns shared cross-dataset medical semantics while confining source-specific variation to lightweight residual experts in deeper decoder layers. To further exploit clean supervision from labeled datasets, we train in a unified disease vocabulary with masked multi-dataset supervision, enabling the model to leverage complementary annotations without introducing false negatives. Experiments on large-scale chest X-ray benchmarks show that HarMoE consistently improves zero-shot classification, out-of-distribution transfer, and grounding over strong baselines. Our results suggest that building robust radiology VLMs requires moving beyond single-source image-report alignment toward structured knowledge construction from heterogeneous datasets with cleaner supervision and broader coverage. Code and the 873k harmonized dataset will be released at https://github.com/Roypic/harmoe.

Problem

• chest radiograph
• multi-source learning
• dataset heterogeneity
• vision-language model
• medical semantics

Innovation

• Mixture-of-Experts
• multi-source pretraining
• dataset disentanglement
• vision-language model
• chest radiograph