MoPET: Parameter-Efficient Mixture-of-Experts for Unified Medical Image Classification

2026-07-31

AI Summary

This work addresses the challenges of parameter-efficient fine-tuning (PEFT) in medical image classification—specifically, model isolation, lack of unification, and negative cross-domain transfer caused by clinical data heterogeneity. To overcome these issues, the authors propose MoPET, which integrates sparse mixture-of-experts (MoE) architecture with PEFT for the first time. MoPET embeds multiple low-rank expert modules into a frozen pretrained backbone and employs a learnable sparse routing mechanism to dynamically select a subset of experts for each input, thereby balancing capacity sharing and gradient conflict mitigation. Evaluated on MedMNIST, MoPET unifies four heterogeneous tasks with an average accuracy of 93.46%, outperforming individual PEFT models. When augmented with auxiliary data co-training, it further boosts performance on data-scarce tasks to 83.58%, substantially alleviating negative transfer and enhancing generalization.

Abstract

Adapting deep learning models to profound clinical heterogeneity typically relies on parameter-efficient fine-tuning (PEFT) to avoid the severe overfitting associated with full end-to-end network updates. Although PEFT successfully navigates limited data scenarios, it inherently forces the training of a separate, isolated adapter for every specific diagnostic task. Consolidating these isolated adapters into a single generalist network risks negative transfer, as optimization gradients from conflicting visual domains interfere. To address this, we propose MoPET, a mixture-of-experts (MoE) method that uses a learned sparse router to direct each input through a small subset of low-rank PEFT experts injected into a frozen foundation model, sharing capacity across datasets while limiting cross-domain gradient conflict. Through selected evaluations on the MedMNIST benchmark, we first establish that PEFT outperforms full network updates, improving average accuracy from 86.50% to 88.97%. We then show that a single MoPET model consolidates four heterogeneous datasets into one network, improving average accuracy over the best isolated PEFT adapters (93.46% versus 92.83%). Finally, we show that co-training with auxiliary datasets improves accuracy on data-constrained clinical targets, raising average target accuracy over the strongest isolated adapter from 81.58% to 83.58%. Our source code is publicly available at https://github.com/sdoerrich97/mopet .

Problem

• medical image classification
• clinical heterogeneity
• parameter-efficient fine-tuning
• negative transfer
• mixture-of-experts

Innovation

• Parameter-Efficient Fine-Tuning
• Mixture-of-Experts
• Medical Image Classification
• Sparse Routing
• Foundation Model Adaptation