Rethinking Pairwise Token Interaction in Spiking Transformers

2026-09-22

AI Summary

研究提出Gated Spike Axial Propagation机制，解决脉冲变换器中基于稀疏二进制表示的长程通信问题，通过传播-选择过程实现结构化信息传递。

Abstract

Spiking Transformers inherit token interaction mechanisms from conventional Transformers, yet their sparse binary representations fundamentally alter how token-to-token communication is established. In particular, spike-based query-key matching produces highly sparse and input-dependent interaction patterns, coupling information propagation to the instantaneous availability of matching spike events. This motivates a different interaction paradigm in which long-range communication does not rely solely on pairwise spike coincidence. We therefore propose Gated Spike Axial Propagation (GSAP), a spike-native token interaction mechanism that decouples information propagation from context selection. Instead of directly determining communication through query-key matching, GSAP first propagates spike-based context along the horizontal and vertical axes, allowing information to reach distant tokens through structured sequential propagation. A receiver-conditioned gate then determines how much of the propagated context is incorporated at each token, while a lightweight local pathway preserves fine-grained neighborhood information. In this way, GSAP reformulates token interaction as a propagate-then-select process, enabling structured long-range communication while retaining the sparse event-driven nature of spiking representations. Code is available at https://github.com/Fancyssc/GSAP.

Problem

• Spiking Transformers
• token interaction
• sparse binary representations
• query-key matching
• information propagation

Innovation

• Gated Spike Axial Propagation
• spiking transformers
• structured long-range communication
• sparse event-driven