FSTA-SNN:Frequency-based Spatial-Temporal Attention Module for Spiking Neural Networks

2024-12-15

AI Summary

Spiking Neural Networks (SNNs) hold great promise for energy-efficient neuromorphic computing, yet their intermediate spike representations are often neglected—leading to weak spatiotemporal feature encoding, severe spike redundancy, and compromised accuracy and efficiency. To address this, we propose Frequency-based Spatiotemporal Attention (FSTA), the first attention mechanism explicitly modeling spike dynamics in the frequency domain. FSTA jointly optimizes spiking behavior through three complementary components: spectral modeling of spike trains, directional spatial selection, and temporal consistency regularization—enabling fine-grained, low-overhead redundancy suppression. As a lightweight, plug-and-play module, FSTA seamlessly integrates with both ANN-to-SNN conversion and direct SNN training paradigms. Evaluated across multiple benchmarks, FSTA achieves up to 42% average spike rate reduction while improving classification accuracy—outperforming state-of-the-art SNN methods and substantially unlocking SNNs’ representational capacity and energy efficiency.

Technology Category

• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems
• Machine Learning: Learning on the Edge & Model Compression

Application Category

• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Spiking Neural Networks (SNNs) are emerging as a promising alternative to Artificial Neural Networks (ANNs) due to their inherent energy efficiency. Owing to the inherent sparsity in spike generation within SNNs, the in-depth analysis and optimization of intermediate output spikes are often neglected. This oversight significantly restricts the inherent energy efficiency of SNNs and diminishes their advantages in spatiotemporal feature extraction, resulting in a lack of accuracy and unnecessary energy expenditure. In this work, we analyze the inherent spiking characteristics of SNNs from both temporal and spatial perspectives. In terms of spatial analysis, we find that shallow layers tend to focus on learning vertical variations, while deeper layers gradually learn horizontal variations of features. Regarding temporal analysis, we observe that there is not a significant difference in feature learning across different time steps. This suggests that increasing the time steps has limited effect on feature learning. Based on the insights derived from these analyses, we propose a Frequency-based Spatial-Temporal Attention (FSTA) module to enhance feature learning in SNNs. This module aims to improve the feature learning capabilities by suppressing redundant spike features.The experimental results indicate that the introduction of the FSTA module significantly reduces the spike firing rate of SNNs, demonstrating superior performance compared to state-of-the-art baselines across multiple datasets.

Problem

• Enhance feature learning in Spiking Neural Networks
• Reduce spike firing rate for energy efficiency
• Optimize spatial-temporal feature extraction in SNNs

Innovation

• Frequency-based Spatial-Temporal Attention Module
• Suppressing redundant spike features
• Enhancing Spiking Neural Networks efficiency