LowRank-SSM: Hardware-Software Co-Design for Rank-Reduced Mamba Acceleration on FPGA

2026-08-03

AI Summary

This work addresses the high memory and bandwidth overheads imposed by the large input/output projection weights of Mamba-style state space models when deployed on FPGAs. To overcome this challenge, the authors propose a hardware-software co-optimization framework: on the software side, they employ truncated singular value decomposition (SVD) for weight compression together with a greedy frequency-band rank allocation algorithm; on the hardware side, they design a fully pipelined accelerator that supports cross-layer mixed-rank execution, explicitly treating projection rank as a hardware design variable to systematically balance accuracy and throughput. Integrated optimizations—including dual-path projection, fused selective scan, multi-AXI master DDR4 access, and INT8 quantization—enable the implementation on a Xilinx Versal VC1902 FPGA to achieve 7.89 tokens/s throughput, delivering a 2.19× improvement in throughput and a 2.03× gain in energy efficiency over the current best solution while maintaining comparable accuracy and power consumption.

Abstract

State Space Models(SSMs) such as Mamba and Mamba-2 achieve linear-time autoregressive inference, making them attractive for latency-sensitive and resource-constrained deployment. Yet their large input and output projection layers impose quadratic weight memory and off-chip bandwidth costs that bottleneck practical FPGA deployment, accounting for over 60% per-token runtime at sequence lengths of 1,024 and beyond. Existing accelerators reduce this overhead through quantization or activation sparsity, but none treat projection rank as an explicit hardware design variable, leaving a systematic accuracy-throughput trade-off unexplored. We present LowRank-SSM, a hardware-software co-design framework that closes this gap. On the software side, we decompose the input and output projection weights via post-training truncated SVD and introduce a greedy bandwise rank-allocation algorithm that searches for the per-band rank vector that minimizes weight storage while respecting a user-specified accuracy constraint. On the hardware side, we map the resulting factored projections onto a fully-pipelined accelerator on an FPGA, featuring a dual-path projection(low-rank path and full-rank path), a fused selective-scan unit, and five independent AXI master bundles that saturate DDR4 bandwidth without bus contention. A per-band runtime rank mask enables mixed-rank execution across all 64 layers with zero architectural overhead. On Xilinx Versal VC1902 at 400 MHz, the deployed mixed-rank INT8 design achieves 7.89~tokens/s, representing a ${2.19\times}$ throughput improvement and ${2.03\times}$ energy-efficiency improvement over SOTA at comparable power and accuracy.

Problem

• State Space Models
• FPGA acceleration
• projection rank
• hardware-software co-design
• memory bottleneck

Innovation

• Low-Rank Decomposition
• Hardware-Software Co-Design
• State Space Models
• FPGA Acceleration
• Rank Allocation