SMM Transformer: Leveraging Spiking Neural Networks for Multimodal Tasks

2026-08-02

AI Summary

This work addresses the challenges of training instability and incompatibility between dense Softmax attention and spiking communication mechanisms when constructing multimodal Transformers with spiking neural networks (SNNs). To overcome these issues, the authors propose the SMM Transformer framework, which introduces a novel PLMP neuron model, a spiking temporal backpropagation (P-STBP) algorithm, channel-wise co-activation with self-compensation mechanisms, and replaces conventional Softmax with the SMSA spiking attention module. Additionally, an SMoE spiking mixture-of-experts architecture is designed to enable efficient multimodal fusion. The proposed method achieves accuracy on par with artificial neural networks across multiple vision and multimodal benchmarks, while the SMSA module reduces attention computation energy consumption by up to 97% at the operator level, significantly enhancing energy efficiency.

Abstract

Spiking Neural Networks (SNNs) enable event-driven computation with sparse activations, but building multimodal Transformers on SNNs is hindered by unstable training in deep spiking stacks and the mismatch between dense softmax attention and spike-based communication. We propose SMM Transformer, an SNN-based multimodal Transformer framework that combines (i)PLMP, a Parallel LIF with Multistage Learnable Parameters neuron and a tailored P-STBP algorithm for stable deep SNN training, (ii) SMSA, an attention-inspired spike-driven token-mixing module that replaces dense pairwise softmax attention with channel-wise spike co-activation and self-compensation, and (iii)SMoE, a spiking mixture-of-experts module for modality-aware fusion. Across visual and multimodal benchmarks, SMM Transformer achieves competitive accuracy compared to ANN baselines. Under a standard MAC/AC arithmetic model, SMSA reduces the estimated operator-level compute energy of the attention module by up to 97%, while whole-model profiling shows more moderate but consistent efficiency gains.

Problem

• Spiking Neural Networks
• Multimodal Transformer
• Spike-based Communication
• Unstable Training
• Attention Mechanism

Innovation

• Spiking Neural Networks
• Multimodal Transformer
• Spike-based Attention
• Energy Efficiency
• Mixture of Experts