PrimeSVT: An Automated Memory-aware Pruning Framework with Prioritized Compression Policy for Spiking Vision Transformers

2026-06-02

AI Summary

This work addresses the challenge of deploying Spiking Vision Transformers (SViTs) on embedded devices due to their large model size and the limitations of existing unstructured pruning methods, which rely on specialized hardware and extensive manual tuning. To overcome these issues, we propose PrimeSViT, the first framework enabling automated structured pruning for SViTs. PrimeSViT introduces a joint ranking of layers based on parameter count and robustness, followed by a coarse-to-fine channel-level filter pruning strategy guided by L2-norm criteria under user-specified accuracy and memory constraints. Without requiring specialized hardware or fine-tuning, our method retains 70.3% accuracy; with fine-tuning, it achieves 72.9% (compared to the original 73.3%) while reducing memory usage by 26.68%, substantially enhancing the feasibility of embedded deployment.

Abstract

The large sizes of Spiking Vision Transformers (SViTs) still hinder their embedded implementation, highlighting the need for model compression. State-of-the-art works compress SViT models through unstructured pruning, which needs specialized hardware accelerators for their specific sparsity patterns to maximize efficiency gains. Moreover, their manual approach requires a huge design time to find an appropriate pruning setting for each network, thus making this approach not scalable. To address this limitation, we propose PrimeSVT, a novel framework that performs automated memory-aware structured pruning on pre-trained SViT models, thereby maximizing their efficiency gains during inference amenable to widely-used computing architectures. To achieve this, PrimeSVT first sorts the SViT layers based on their sizes (i.e., number of parameters), identifies the targeted pruning layers based on their robustness under different pruning rates, then leverages this order for compressing the model layer-by-layer sequentially from the largest one to the smallest one (i.e., so-called prioritized compression policy), while considering the user-defined constraints (i.e., acceptable accuracy and memory saving). In each layer, PrimeSVT employs channel-wise filter pruning based on their L2-norm values to structurally remove the non-significant weights. Experimental results show that PrimeSVT saves 26.68% memory through automated single-shot pruning, while preserving accuracy within 3% (70.3% without fine-tuning and 72.9% with fine-tuning) from the original unpruned SViT model (73.3%), thus meeting the accuracy and memory constraints. These show that our PrimeSVT framework enables design automation for SViTs and their embedded implementation.

Problem

• Spiking Vision Transformers
• model compression
• structured pruning
• embedded implementation
• memory efficiency

Innovation

• structured pruning
• memory-aware
• prioritized compression policy
• Spiking Vision Transformers
• automated model compression