Problem
Research questions and friction points this paper is trying to address.
Spiking Neural Networks
Second-order optimization
Loss landscape
Training stability
Fisher information matrix
On the second-order optimization for spiking neural networks
🤖 AI Summary
This study addresses the challenge that spiking neural networks (SNNs) exhibit steep loss landscapes due to sparse binary activations, rendering conventional diagonal curvature optimizers inadequate for capturing their complex geometric properties. To this end, we propose SpiKFAX, which introduces Kronecker-Factored Approximate Curvature (K-FAC) second-order optimization into SNN training for the first time. By accounting for the sparsity, discreteness, and temporal recurrent dynamics inherent in SNNs, our method constructs a computationally tractable approximation of the Fisher information matrix, effectively overcoming the failure modes of existing optimizers. Experimental results demonstrate that SpiKFAX significantly improves test accuracy and training stability across five architectures and seven datasets.
📝 Abstract
Spiking Neural Networks (SNNs) offer an energy-efficient alternative to conventional neural networks by exploiting sparse, binary spikes, and event-driven computation. However, the training of SNNs remains challenging, as spiking activations create a sharp loss landscape that hinders training, and diagonal-curvature optimizers such as the Adam family may fail to capture this geometry. The extension of curvature-based optimization methods to SNNs is further complicated by the sparse, discrete, and temporally recurrent nature of their underlying dynamics. To address these limitations, we propose SpiKFAX, a second-order optimization method that formulates a computationally tractable, Kronecker-factored approximation of the Fisher information matrix specifically adapted to the structure of SNNs. Empirical evaluation across five architectures and seven datasets demonstrates that SpiKFAX consistently yields improvements in test accuracy and training stability relative to other popular optimizers.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Spiking Neural Networks
Second-order optimization
Kronecker-factored approximation
Fisher information matrix
SpiKFAX
🔎 Similar Papers
No similar papers found.
