ASTER: Attention-based Spiking Transformer Engine for Event-driven Reasoning

2025-11-10

AI Summary

Spiking Transformer models—hybrid architectures combining spiking neural networks (SNNs) and Transformers—exhibit low energy efficiency on general-purpose hardware, while existing neuromorphic or processing-in-memory (PIM) accelerators struggle to handle their high spatio-temporal sparsity and complex operations. Method: This paper proposes a hardware acceleration engine for event-driven visual inference, featuring an analog-digital hybrid PIM architecture, spatio-temporally sparse-aware dataflow, dynamic optimization mechanisms (including layer skipping and timestep reduction), and a Bayesian optimization–driven algorithm–microarchitecture co-design methodology. Contribution/Results: Evaluated on ImageNet, CIFAR-10 DVS, and DVSGesture benchmarks, the accelerator achieves up to 467× and 1.86× higher energy efficiency than edge GPUs and state-of-the-art PIM accelerators, respectively, while maintaining state-of-the-art accuracy.

Technology Category

Cognitive Modeling & Cognitive Systems: Neural Spike CodingMachine Learning: Hardware-aware MLComputer Vision: Multi-modal Vision

Application Category

Search and Retrieval-Augmented AI: Efficiency and scalability of Web search enginesGraph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphsSecurity and Privacy: Privacy-enhancing technologies

Abstract

The integration of spiking neural networks (SNNs) with transformer-based architectures has opened new opportunities for bio-inspired low-power, event-driven visual reasoning on edge devices. However, the high temporal resolution and binary nature of spike-driven computation introduce architectural mismatches with conventional digital hardware (CPU/GPU). Prior neuromorphic and Processing-in-Memory (PIM) accelerators struggle with high sparsity and complex operations prevalent in such models. To address these challenges, we propose a memory-centric hardware accelerator tailored for spiking transformers, optimized for deployment in real-time event-driven frameworks such as classification with both static and event-based input frames. Our design leverages a hybrid analog-digital PIM architecture with input sparsity optimizations, and a custom-designed dataflow to minimize memory access overhead and maximize data reuse under spatiotemporal sparsity, for compute and memory-efficient end-to-end execution of spiking transformers. We subsequently propose inference-time software optimizations for layer skipping, and timestep reduction, leveraging Bayesian Optimization with surrogate modeling to perform robust, efficient co-exploration of the joint algorithmic-microarchitectural design spaces under tight computational budgets. Evaluated on both image(ImageNet) and event-based (CIFAR-10 DVS, DVSGesture) classification, the accelerator achieves up to ~467x and ~1.86x energy reduction compared to edge GPU (Jetson Orin Nano) and previous PIM accelerators for spiking transformers, while maintaining competitive task accuracy on ImageNet dataset. This work enables a new class of intelligent ubiquitous edge AI, built using spiking transformer acceleration for low-power, real-time visual processing at the extreme edge.

Problem

• Optimizing hardware for spiking transformers' binary spike-driven computation
• Addressing memory inefficiency from high sparsity in event-driven models
• Enabling low-power real-time visual reasoning on edge devices

Innovation

• Hybrid analog-digital PIM architecture for spiking transformers
• Custom dataflow minimizing memory access under sparsity
• Bayesian optimization for algorithmic-hardware co-exploration