LongSpike: Fractional Order Spiking State Space Models for Efficient Long Sequence Learning

2026-06-11

AI Summary

This work addresses the limited long-term memory capacity of conventional spiking neural networks (SNNs), which are typically modeled using first-order differential equations and thus struggle to capture complex dependencies in long sequences. To overcome this limitation, the study introduces fractional-order control theory into SNNs for the first time, proposing a neuron state-space model grounded in fractional calculus. This formulation endows the network with intrinsic long-range memory while enabling efficient parallel training through a state-space representation. The approach preserves the inherent sparsity and energy efficiency of SNNs yet substantially enhances their ability to model long sequences. Empirical evaluations on benchmarks including LRA, WikiText-103, and Speech Commands demonstrate superior performance over state-of-the-art SNN models, achieving a better trade-off between accuracy and energy efficiency.

Abstract

Spiking Neural Networks (SNNs) are well-regarded for their biological plausibility and energy efficiency in processing sequential data. However, dominant SNN architectures typically rely on first-order Ordinary Differential Equations (ODEs) to govern neuronal state transitions. This first-order assumption imposes a "memoryless" bottleneck, limiting the model's capacity to capture the complex, long-range dependencies inherent in long-sequence tasks. In this work, we propose LongSpike, a novel SNN framework that integrates fractional-order State-Space Modeling, or f-SSM, from control theory into the spiking domain. By extending traditional integer-order SSMs to the fractional-calculus regime, LongSpike enables the hierarchical integration of neuronal dynamics with long-memory kernels. To mitigate the computational overhead and parallelization challenges typically associated with fractional operators, we leverage a state-space formulation that supports efficient, parallel training. Empirical evaluations on challenging benchmarks, including Long Range Arena (LRA), large-scale WikiText-103, and Speech Commands, demonstrate that LongSpike outperforms state-of-the-art SNNs in accuracy while preserving sparse synaptic computation. The code is available at https://github.com/xinruihe389-commits/LongSpike.

Problem

• Spiking Neural Networks
• long-range dependencies
• fractional-order
• state-space models
• memory bottleneck

Innovation

• Spiking Neural Networks
• Fractional-order Systems
• State-Space Models
• Long-range Dependencies
• Efficient Training