Rethinking Expert Training for Model Merging with Prompt Learning

2026-07-27

AI Summary

This work addresses the limitation of existing model merging methods, which typically assume that experts are obtained via full-parameter fine-tuning and thereby overlook how training strategies affect merge compatibility, often leading to inter-task interference. To mitigate this, the authors propose Dual-Tuned Experts (DTEs), a two-stage training strategy that first learns reusable prompts across tasks and then fine-tunes only the vision encoder while keeping the backbone frozen. This approach is the first to integrate prompt learning into expert training, decoupling prompt adaptation from backbone updates to substantially reduce task-specific perturbations. As a result, DTEs enhance both the stability and performance of merging heterogeneous experts. Experiments demonstrate that DTEs consistently outperform conventional fully fine-tuned experts across diverse CLIP architectures and expert types, while remaining compatible with mainstream model merging techniques.

Abstract

Model merging aims to combine multiple domain-specialized experts trained from a shared foundation model into a single multi-task model. Existing approaches largely focus on improving the merging procedure itself and typically assume experts obtained through full-parameter fine-tuning. In this work, we revisit expert training for model merging. We first show that prompt-based adaptation provides a strong baseline: independently learned prompts can be exploited across tasks while keeping the backbone fixed, avoiding the interference introduced by weight merging. Building on this observation, we introduce Dual-Tuned Experts (DTEs), a two-stage training strategy that first learns prompts and then fine-tunes the vision encoder. This reduces the magnitude of task-specific parameter updates and produces experts with higher merge compatibility. Experiments across multiple CLIP architectures, full fine-tuning, and LoRA experts show that DTEs consistently improve merged performance of standard merging approaches and remain effective even when combining heterogeneous sets of experts.

Problem

• model merging
• expert training
• prompt learning
• multi-task learning
• merge compatibility

Innovation

• model merging
• prompt learning
• Dual-Tuned Experts
• multi-task learning
• foundation model adaptation