Theory on Mixture-of-Experts in Continual Learning

2024-06-24

arXiv.org

AI Summary

Existing mixture-of-experts (MoE) models for continual learning lack rigorous theoretical foundations. Method: This paper establishes the first theoretical analysis framework for MoE in continual learning, based on overparameterized linear regression. It derives explicit closed-form expressions for both forgetting error and generalization error, characterizes how expert specialization and dynamic routing jointly mitigate catastrophic forgetting, proves that gating networks must be selectively frozen to ensure convergence, and quantifies the trade-off between the number of experts and convergence iterations. Results: The theory demonstrates that MoE strictly outperforms single-expert models. Extensive experiments on synthetic and real-world benchmarks with deep neural networks empirically validate the theoretical predictions, confirming the efficacy of expert specialization, optimal gating freezing schedules, and the expert-count–convergence trade-off. This work provides the first interpretable, theoretically grounded foundation for MoE-based continual learning.

Technology Category

Machine Learning: Mixture of Experts (MoE)Search and Optimization: Learning to SearchNatural Language Processing: Learning & Optimization for NLP

Application Category

User Modeling, Personalization and Recommendation: Practical large-scale studies of user experienceSearch and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for rankingEconomics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI

Abstract

Continual learning (CL) has garnered significant attention because of its ability to adapt to new tasks that arrive over time. Catastrophic forgetting (of old tasks) has been identified as a major issue in CL, as the model adapts to new tasks. The Mixture-of-Experts (MoE) model has recently been shown to effectively mitigate catastrophic forgetting in CL, by employing a gating network to sparsify and distribute diverse tasks among multiple experts. However, there is a lack of theoretical analysis of MoE and its impact on the learning performance in CL. This paper provides the first theoretical results to characterize the impact of MoE in CL via the lens of overparameterized linear regression tasks. We establish the benefit of MoE over a single expert by proving that the MoE model can diversify its experts to specialize in different tasks, while its router learns to select the right expert for each task and balance the loads across all experts. Our study further suggests an intriguing fact that the MoE in CL needs to terminate the update of the gating network after sufficient training rounds to attain system convergence, which is not needed in the existing MoE studies that do not consider the continual task arrival. Furthermore, we provide explicit expressions for the expected forgetting and overall generalization error to characterize the benefit of MoE in the learning performance in CL. Interestingly, adding more experts requires additional rounds before convergence, which may not enhance the learning performance. Finally, we conduct experiments on both synthetic and real datasets to extend these insights from linear models to deep neural networks (DNNs), which also shed light on the practical algorithm design for MoE in CL.

Problem

• Theoretical analysis of Mixture-of-Experts in Continual Learning.
• Mitigating catastrophic forgetting via expert diversification.
• Impact of expert count on convergence and performance.

Innovation

• Mixture-of-Experts mitigates forgetting
• Theoretical analysis via overparameterized regression
• Gating network update termination for convergence