Efficient Distillation of Deep Spiking Neural Networks for Full-Range Timestep Deployment

2025-01-27

AI Summary

To address the low accuracy, rigid temporal deployment (relying on fixed inference timesteps), and poor generalization across varying timesteps in Spiking Neural Networks (SNNs), this paper proposes the first universal knowledge distillation framework supporting arbitrary inference timesteps. Methodologically, it integrates spatiotemporal-aware distillation, implicit multi-timestep modeling, and joint SNN–ANN optimization. Crucially, it establishes the first theoretical guarantee of implicit model convergence across the full range of timesteps. Experiments demonstrate state-of-the-art accuracy among distilled SNNs on CIFAR-10/100, CIFAR10-DVS, and ImageNet. Moreover, the framework significantly improves energy efficiency and plug-and-play deployment flexibility—enabling seamless adaptation to diverse timesteps without retraining.

Technology Category

• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems
• Computer Vision: Learning & Optimization for CV

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments

Abstract

Spiking Neural Networks (SNNs) are emerging as a brain-inspired alternative to traditional Artificial Neural Networks (ANNs), prized for their potential energy efficiency on neuromorphic hardware. Despite this, SNNs often suffer from accuracy degradation compared to ANNs and face deployment challenges due to fixed inference timesteps, which require retraining for adjustments, limiting operational flexibility. To address these issues, our work considers the spatio-temporal property inherent in SNNs, and proposes a novel distillation framework for deep SNNs that optimizes performance across full-range timesteps without specific retraining, enhancing both efficacy and deployment adaptability. We provide both theoretical analysis and empirical validations to illustrate that training guarantees the convergence of all implicit models across full-range timesteps. Experimental results on CIFAR-10, CIFAR-100, CIFAR10-DVS, and ImageNet demonstrate state-of-the-art performance among distillation-based SNNs training methods.

Problem

• Simplified Spiking Neural Networks
• Time-length Flexibility
• Model Efficiency

Innovation

• Spike Neural Networks
• Adaptive Timing Framework
• State-of-the-art Performance