🤖 AI Summary
Dynamic routing in Mixture-of-Experts (MoE) models is vulnerable to redundant neuron activations, leading to biased expert selection and insufficient expert diversity.
Method: We propose a neural suppression–enhanced dynamic routing mechanism that applies learnable suppression signals to redundant neuron populations within the shared feature space prior to routing decisions, explicitly attenuating collinear responses to improve discriminability and specialization of expert path selection.
Contribution/Results: Unlike prior MoE approaches, this work is the first to systematically demonstrate the routing-quality benefits of neural suppression and integrate it end-to-end into Transformer-like architectures without increasing parameter count. Experiments across multiple NLP and vision benchmarks show consistent improvements: +1.2–2.8% accuracy gains, −37% reduction in routing variance, and enhanced expert utilization balance and task adaptability—establishing a novel paradigm for efficient, diverse sparse modeling.
📝 Abstract
To be effective, efficient, and diverse, deep learning models need to dynamically choose its architecture based on signals from a population of neurons. We hypothesize dynamic routing models can be improved with neural inhibition in those neural populations. This means signals commonly shared among the various modes of data statistics can be inhibited so that the routing model can choose a specialized expert path for each data sample. Only through inhibition is the routing mechanism able to effectively select neural pathways. We believe this is an under-studied and under-verified implementation methodology for Mixture-of-Experts, dynamic routing, and transformer language models. We provide experimental evidence that the neural inhibition algorithm significantly boosts the performance of general tasks and motivates more effort to be invested in this research direction.