SpikeDecoder: Realizing the GPT Architecture with Spiking Neural Networks

2026-06-10

AI Summary

This work addresses the high energy consumption of conventional Transformers in natural language processing, which limits their applicability in energy-constrained scenarios. The study proposes the first fully spiking neural network (SNN)-based Transformer decoder tailored for NLP tasks, introducing a trainable SNN decoder architecture, SNN-compatible normalization techniques, and multiple text-to-spike embedding strategies. The authors systematically evaluate the impact of module substitution, residual connections, and normalization on model performance. By extending SNN Transformers beyond vision-only applications or encoder-only designs, the proposed model maintains functional equivalence to its artificial neural network (ANN) counterpart while achieving a theoretical energy reduction of 87%–93% compared to standard ANN baselines.

Abstract

The Transformer architecture is widely regarded as the most powerful tool for natural language processing, but due to a high number of complex operations, it inherently faces the issue of high energy consumption. To address this issue, we consider Spiking Neural Networks (SNNs), which are an energy-efficient alternative to conventional Artificial Neural Networks (ANNs) due to their naturally event-driven approach to processing information. However, this inherently makes them difficult to train. Often, many SNN-based models circumvent this issue by converting pre-trained ANNs. More recently, attempts have been made to design directly trainable SNN-based adaptations of the Transformer model structure. Although the results showed great promise, the application field was computer vision. Moreover, the proposed model incorporates only encoder blocks. In this paper, we propose SpikeDecoder, a fully SNN-based implementation of the Transformer decoder block, for applications in natural language processing. In a series of experiments, we analyze the impact of exchanging different blocks of the ANN model with spike-based alternatives to identify trade-offs and significant sources of performance loss. We further investigate the role of residual connections and the selection of SNN-compatible normalization techniques. Besides the work on the model architecture, we formulate and compare different embedding methods to project text data into spikes. Finally, we demonstrate that our proposed SNN-based decoder block reduces the theoretical energy consumption by 87% to 93% compared to the ANN baseline.

Problem

• Spiking Neural Networks
• Transformer
• Energy Efficiency
• Natural Language Processing
• Decoder

Innovation

• Spiking Neural Networks
• Transformer Decoder
• Energy Efficiency
• Natural Language Processing
• SNN Embedding