STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization

2026-09-29

AI Summary

This study addresses the severe accuracy degradation caused by quantizing recurrent states in linear attention mechanisms, proposing a spatiotemporal-aware post-training quantization framework. The method reveals, for the first time, a complementary relationship between quantization errors along temporal persistence and spatial critical rows, leveraging this insight to dynamically allocate bit-widths while jointly fitting row- and column-wise scaling factors to optimize state distributions. By integrating Delta-rule recurrent states, SGLang compatibility, and GPU kernel optimizations, the proposed approach achieves near-FP32 accuracy under 6-bit configurations and outperforms INT8 baselines at 4-bit precision. Ultimately, it realizes over 5× model compression while reducing serving memory overhead by 68.7%.

Abstract

Linear attention replaces growing KV caches with fixed-size recurrent states, yet these persistent states can become a substantial memory bottleneck under concurrent serving. Directly quantizing recurrent states to low precision often leads to severe accuracy degradation, as quantization errors propagate through successive state updates. We discover that the impact of these errors depends on two complementary dimensions: temporally, errors in long-lived memory can persist across many decoding steps; spatially, errors in different key rows affect model outputs differently, while state magnitudes vary substantially along both rows and columns. Motivated by these observations, we propose STEPQuant, a spatial-temporal post-training quantization framework for Delta-rule recurrent states. STEPQuant allocates precision according to error magnitude and memory lifetime, and jointly fits key-row and value-column scales based on state distributions and key-row impact on output error. Experiments on Qwen3.8-27B and Kimi-Linear-48B-A3B-Instruct across both long- and short-generation benchmarks show that STEPQuant closely matches FP32-state accuracy under a nominal 6-bit budget and outperforms uniform INT8 in its 4-bit configuration. Integrated into SGLang with optimized GPU kernels, 6-bit STEPQuant achieves over 5x recurrent-state compression and reduces total serving memory by up to 68.7%. Our code is available at https://github.com/Dreamer-Toby/STEPQuant.

Problem

• Linear Attention
• Recurrent State Quantization
• Memory Bottleneck
• Error Propagation
• Concurrent Serving

Innovation

• Post-training quantization
• Linear attention
• Delta-rule recurrent states
• Spatial-temporal error analysis
• Memory-efficient serving