On Design Principles for Efficient Heterogeneous DRAM-PIM-GPU Systems

2026-08-04

AI Summary

This study addresses the significant overestimation—up to 3.85×—of energy efficiency in large language model (LLM) inference on heterogeneous DRAM-PIM-GPU systems, which commonly neglect static power consumption, channel configuration, and workload mapping. Through full-system simulation and empirical validation using models such as OPT and Mamba2 across varying batch sizes and input/output lengths, this work reveals that static power dominates real-world energy efficiency, that performance scales monotonically with memory channel count, and that near-optimal memory hierarchies share common structural traits. The authors propose a holistic co-design principle for system-wide optimization, demonstrating its necessity in long-output generation scenarios and showing that judicious workload mapping can improve end-to-end energy efficiency by up to 5.6%.

Abstract

Heterogeneous DRAM-based processing-in-memory (PIM)-GPU systems promise significant efficiency gains for decode-phase large language model (LLM) inference, particularly in long-output generation, yet current design practices overlook critical factors that determine real-world performance. Through systematic evaluation of diverse architectures and workloads (OPT-7B/70B, Mamba2-2.7B/70B), we reveal three fundamental design principles: (i) static power consumption (DRAM leakage, refresh, and GPU idle power) can dominate the efficiency calculus, causing dynamic-only models to overestimate tokens/s/W by up to 3.85X for realistic deployments (Mamba2-2.7B, batch size 1, 128 input tokens, and 2,048 output tokens); (ii) decoding performance is monotonically non-decreasing with channel count across all evaluated models and workloads, generally plateauing at high channel counts for low-batch workloads; under a fixed-capacity sweep, all models instead share a common near-optimal hierarchy configuration, with substantially larger misconfiguration penalties for attention-based models; (iii) workload mapping strategies provide bounded improvements (up to 14.0%/17.4% kernel-level latency/energy reduction, up to 5.6% end-to-end gain) and are not primary bottlenecks. Significant efficiency gains require system-wide co-optimization. These principles provide design-space guidance for architects designing the next generation of memory-accelerated LLM systems.

Problem

• DRAM-PIM-GPU
• LLM inference
• energy efficiency
• system design
• decode-phase

Innovation

• Processing-in-Memory
• Heterogeneous Architecture
• LLM Inference
• Static Power Optimization
• Memory Hierarchy Co-design