🤖 AI Summary
This study addresses the absence of routing mechanisms and the challenge of incomplete multimodal inputs in integrating Spiking Neural Networks (SNNs) with Mixture-of-Experts (MoE) models by proposing the SpikeMoE framework. Methodologically, it introduces a hippocampus-inspired spiking k-WTA routing mechanism that leverages lateral inhibition and refractory periods to achieve neuron-level dynamic expert selection. Additionally, a two-stage missing-modality modeling module is constructed to enhance robustness. This work achieves state-of-the-art SNN performance across visual, linguistic, and multimodal benchmarks, rivaling Artificial Neural Networks (ANNs) while substantially improving energy efficiency. These results validate the inherent advantages of sparse computation within neuromorphic architectures.
📝 Abstract
Spiking Neural Networks (SNNs) enable event-driven computation through biologically inspired dynamics at the neuronal scale, while Mixture-of-Experts (MoE) perform conditional computation through expert selection at the model scale. Integrating their strengths offers potential for flexible neural architectures. A key challenge, however, lies in designing an expert selection mechanism based on spiking activity. To address this, we introduce a spike-based k-WTA Router inspired by competition-inhibition observed in the hippocampal CA1 region. The router incorporates lateral inhibition and refractory period to select Top-K experts according to discrete spike counts. Building on this, we present SpikeMoE, a framework that integrates neuronal-scale spiking dynamics with model-scale expert selection. To address incomplete multisensory inputs in multimodal tasks, we further equip SpikeMoE with a two-stage missing-modality modeling module that combines empirical prototypes from an observed-modality pool with modality-specific learnable embeddings to construct missing-modality representations. Experiments on vision, language, and multimodal benchmarks demonstrate that SpikeMoE achieves state-of-the-art performance among the SNN baselines, matches or exceeds the performance of ANN counterparts, and maintains robustness across diverse missing-modality conditions. These results demonstrate a favorable trade-off between performance and energy efficiency, validating the integration of spiking dynamics with sparse expert computation and highlighting SpikeMoE as a promising approach to energy-efficient brain-inspired computing.