Spike-HTR: Spiking Neural Transformer for Handwritten Text Recognition

2026-08-02

AI Summary

This work addresses the computational inefficiency of spiking neural networks (SNNs) in handwritten text recognition, primarily caused by redundant image backgrounds and dominant blank sequences. To overcome this, the authors propose Spike-HTR, the first framework integrating SNNs with Transformers for this task. It employs InkCoder to encode static images into coarse-to-fine spike streams and introduces a CTC-guided length compression mechanism that drastically reduces redundant temporal computations. Remarkably, the model achieves training and decoding in only two time steps, eliminating the need for external language models or lexicons. Evaluated on IAM, LAM, and READ2016 datasets, Spike-HTR attains character error rates of 3.5/5.4, 2.3/2.5, and 4.2/3.9, respectively, demonstrating state-of-the-art performance while enabling highly efficient, short-time-domain spiking inference.

Abstract

Handwritten Text Recognition (HTR) is computationally imbalanced in two ways: most image pixels are background, and many width-axis sequence positions are blank-dominated. This creates a mismatch for Spiking Neural Networks (SNNs): handwriting is observed as a static image, whereas spiking computation unfolds over timesteps. We propose Spike-HTR, a hybrid spiking recognizer that controls both the number of spiking steps and the number of width positions processed by the deep sequence mixer. To make a static image suitable for short-horizon spiking inference, InkCoder converts it into a coarse-to-fine input stream, where early steps cover broad stroke regions and later steps emphasize sharper stroke details. To reduce sequence computation, a CTC-guided length reducer keeps likely character or uncertain positions and compresses long blank-dominated stretches before deep mixing. With $T{=}2$, Spike-HTR trains only on target data, decodes without language models or lexicons, and reaches validation/test CERs of 3.5/5.4, 2.3/2.5, and 4.2/3.9 on IAM, LAM, and READ2016. Codes are available at https://github.com/QomolangmaH/SpikeHTR.

Problem

• Handwritten Text Recognition
• Spiking Neural Networks
• computational imbalance
• static image
• temporal mismatch

Innovation

• Spiking Neural Networks
• Handwritten Text Recognition
• InkCoder
• CTC-guided Length Reduction
• Hybrid Spiking Transformer