RH+: Row-Hit-Optimized Scheduling for PIM-based LLM Inference

2026-06-03

AI Summary

This work addresses the inefficiency of conventional scheduling in PIM-based autoregressive LLM inference, where performance is bottlenecked by the DRAM row cycle time (nRC), rendering optimizations targeting nCCDAB ineffective for GEMV operations. To overcome this limitation, the authors propose RH+, a novel scheduling method that reengineers the address mapping strategy to co-locate consecutive MAC operations within the same DRAM row. By merely adjusting the access stride, RH+ substantially enhances row locality, circumvents the nRC constraint, and overturns the traditional host-centric interleaving paradigm. Evaluated on an HBM3 PIM architecture using cycle-accurate simulation across four LLM workloads, RH+ achieves 8–12× speedup, over 74% energy reduction, and up to 52× improvement in energy-delay product (EDP).

Abstract

Large language model inference on processing-in-memory (PIM) architectures promises to break the memory wall by performing multiply-accumulate (MAC) operations directly within HBM3 DRAM banks. Prior work identifies the power constraint timing parameter nCCDAB as the primary performance bottleneck and optimizes scheduling accordingly. We demonstrate that for GEMV operations that dominate autoregressive decoding, the DRAM row cycle time (nRC) is 10 to 11 times larger than nCCDAB. Consequently, nCCDAB is entirely masked, rendering prior nCCDAB-focused optimizations ineffective for these workloads. The root cause is inherited host-centric address interleaving, which forces every all-bank MAC command into a different DRAM row. We propose RH+ scheduling, a simple stride change that keeps 32 consecutive MAC operations within the same row. Cycle-accurate simulation across four LLM workloads shows that RH+ delivers 8-12x speedup, over 74% energy reduction, and up to 52x EDP improvement.

Problem

• Processing-in-Memory
• LLM Inference
• DRAM Row Cycle Time
• Address Interleaving
• GEMV

Innovation

• Processing-in-Memory (PIM)
• Row-Hit Optimization
• DRAM Scheduling
• LLM Inference
• nRC Bottleneck