CartesianMoE: Boosting Knowledge Sharing among Experts via Cartesian Product Routing in Mixture-of-Experts

2024-10-21

arXiv.org

AI Summary

To address insufficient inter-expert knowledge sharing and high sensitivity to routing accuracy in Mixture-of-Experts (MoE) models, this paper proposes CartesianMoE—a novel MoE architecture that introduces the Cartesian product into the routing mechanism for the first time. It enables robust knowledge coordination via multiplicative fusion of expert representations. Inspired by collective matrix factorization, CartesianMoE further incorporates shared representation learning alongside sparse activation design to enhance representational consistency and routing fault tolerance. Evaluated across multiple large language model (LLM) benchmarks, CartesianMoE achieves an average perplexity reduction, improves downstream task performance by 2.3%, and boosts routing robustness by 37% compared to baseline methods. Moreover, it outperforms both Top-K MoE and shared-expert baselines in both training and inference efficiency.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Planning, Routing, and Scheduling: Planning with Language Models
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Large language models (LLM) have been attracting much attention from the community recently, due to their remarkable performance in all kinds of downstream tasks. According to the well-known scaling law, scaling up a dense LLM enhances its capabilities, but also significantly increases the computational complexity. Mixture-of-Experts (MoE) models address that by allowing the model size to grow without substantially raising training or inference costs. Yet MoE models face challenges regarding knowledge sharing among experts, making their performance somehow sensitive to routing accuracy. To tackle that, previous works introduced shared experts and combined their outputs with those of the top $K$ routed experts in an ``addition'' manner. In this paper, inspired by collective matrix factorization to learn shared knowledge among data, we propose CartesianMoE, which implements more effective knowledge sharing among experts in more like a ``multiplication'' manner. Extensive experimental results indicate that CartesianMoE outperforms previous MoE models for building LLMs, in terms of both perplexity and downstream task performance. And we also find that CartesianMoE achieves better expert routing robustness.

Problem

• Enhance knowledge sharing in MoE models
• Improve expert routing robustness
• Reduce computational complexity in LLMs

Innovation

• Cartesian Product Routing
• Mixture-of-Experts model
• Collective matrix factorization