Model Merging for Medical LVLMs: A Benchmark and a Winner-Take-All Approach

2026-07-17

AI Summary

This work addresses the high computational and operational overhead of deploying multiple expert medical large vision-language models (LVLMs) by proposing a hyperparameter-free “winner-takes-all” model merging strategy. The approach efficiently integrates LoRA-based expert models by retaining only the most dominant parameters from each, thereby preventing information dilution. To evaluate the method, the authors introduce MergeMedBench—the first benchmark for medical LVLM merging—encompassing eight imaging modalities and sixteen expert models. Experimental results demonstrate that the proposed strategy significantly outperforms existing merging techniques, establishing a strong and efficient baseline for lightweight deployment of medical LVLMs.

Abstract

Large vision-language models (LVLMs) can be adapted to specialized medical imaging tasks via parameter-efficient fine-tuning approaches such as low-rank adaptation (LoRA), leading to a growing ecosystem of expert models tailored to specific imaging modalities and clinical scenarios. However, deploying multiple expert LVLMs in practice incurs substantial computational and operational overhead. Model merging provides a promising solution by consolidating multiple experts into a single model without retraining, yet it remains largely unexplored in the medical domain. In this work, we present the first systematic study of model merging for medical LVLMs. We introduce MergeMedBench, a comprehensive benchmark spanning eight imaging modalities and diverse clinical task types, comprising 16 LoRA fine-tuned models built upon two mainstream architectures. We conduct an extensive evaluation of existing merging methods and further propose winner-take-all, a simple and hyperparameter-free approach that retains only the most dominant parameters across expert models. By preserving the critical parameters that govern model behavior and discarding weaker ones, our method avoids the information dilution inherent in averaging- or alignment-based strategies. Despite its simplicity, winner-take-all consistently outperforms existing approaches, offering both a new perspective on LoRA merging and a strong practical baseline for future research.

Problem

• model merging
• medical LVLMs
• LoRA
• expert models
• deployment overhead

Innovation

• model merging
• medical LVLMs
• LoRA
• winner-take-all
• MergeMedBench