Beyond Global Routing Aggregation: Phase-Aware Expert Merging for MoE Vision-Language Models

2026-08-05

AI Summary

This work addresses the limitations of existing training-free expert merging methods for Mixture-of-Experts vision-language models (MoE-VLMs), which rely on global routing statistics and overlook the distinct routing distributions across different reasoning stages—such as image understanding, question parsing, and answer generation—leading to performance degradation. To overcome this, the authors propose RoleMerge, the first approach that introduces stage-aware expert roles. RoleMerge constructs normalized Routing Role Profiles (RRPs) for each reasoning phase, capturing experts’ functional preferences through phase-normalized routing statistics, and evaluates expert compatibility based on information loss to enable precise, training-free merging. Experiments across three MoE-VLMs and multiple benchmarks demonstrate that RoleMerge significantly outperforms existing methods under identical expert retention rates, achieving up to a 9.6% relative improvement in macro-averaged performance across six tasks.

Abstract

Mixture-of-experts vision-language models (MoE-VLMs) increase model capacity with sparse expert activation, yet deployment requires storing the full expert pool. Training-free expert merging reduces this burden, and many routing-based methods aggregate routing statistics across all tokens to determine merge compatibility. However, MoE-VLM inference is phase-structured: image-context tokens carry visual content, question tokens specify the query, and answer tokens produce the output, with different counts and routing distributions. Because image-context tokens are far more numerous, global aggregation can overemphasize image-context processing and obscure phase-conditioned expert roles, making experts serving different phases appear interchangeable and degrading model performance. We therefore argue that MoE-VLM expert merging should preserve phase-conditioned expert roles, judging compatibility by how experts serve different phases rather than globally aggregated routing statistics. Based on this view, we propose RoleMerge, a training-free method that constructs each expert's Routing Role Profile (RRP) from phase-normalized routing statistics, capturing its relative phase preference. Guided by expert-phase information loss, RoleMerge merges experts with compatible profiles and their corresponding router entries while preserving answer-decoding expert distinctions. Experiments on three models and multiple benchmarks show that RoleMerge preserves more of the full model's performance than alternative expert-merging methods at matched expert-retention ratios, with relative improvements of up to 9.6 percent in six-task macro-average performance. These results validate phase-conditioned expert roles as a more effective basis than global routing aggregation for MoE-VLM expert merging.

Problem

• Mixture-of-experts
• vision-language models
• expert merging
• phase-aware
• routing statistics

Innovation

• expert merging
• phase-aware routing
• Mixture-of-Experts
• vision-language models
• Routing Role Profile