Problem
Research questions and friction points this paper is trying to address.
Large Language Model
Energy Efficiency
On-chip Memory
M3D Integration
Data Movement
LLMET: Enabling Cross-Layer Evaluation of Emerging M3D Memories for Energy-Efficient LLM Serving
🤖 AI Summary
This work addresses the high energy cost of large language model (LLM) inference, primarily caused by limited on-chip cache capacity and frequent accesses to high-bandwidth memory (HBM). The authors propose LLMET, a cross-layer simulation framework, to systematically evaluate—for the first time—the impact of monolithic three-dimensional (M3D) integration on LLM serving energy efficiency through significant on-chip cache expansion. Leveraging M3D back-end-of-line (BEOL) technology, they scale the L2 cache from 40 MB to 1 GB and simulate diverse LLMs and workloads on configurations resembling dual A100 and 8×B200 platforms. Results demonstrate that, for the Llama3.1-70B model, chip energy consumption during the prefill phase is reduced by 44%, with up to 30% savings in decode-phase energy at the edge, underscoring the critical role of large on-chip caches in minimizing data-movement energy.
📝 Abstract
The energy consumption of Large Language Model (LLM) serving is becoming a major system challenge as deployment scales, driven by hardware power and thermal constraints and rising electricity costs. A key contributor to chip energy dissipation is data movement between limited on-chip cache and off-chip High Bandwidth Memory (HBM). Meanwhile, emerging memory technologies such as monolithic 3D (M3D) integration of cache memories at the Back-End-Of-Line (BEOL) of logic chips enable larger and denser on-chip memories, creating new opportunities to reduce costly off-chip traffic. However, it remains unclear whether continuously scaling on-chip memory using emerging technologies can effectively improve the energy efficiency of LLM serving. To address this gap, we develop LLMET (LLM with Emerging Technology), a validated cross-layer simulation framework, and conduct a comprehensive study on the impact of large-capacity on-chip memory technologies across a broad range of models, applications and platforms. Utilizing M3D technology to expand the L2 cache from 40MB to 1GB yields a 44% reduction in chip energy during the Llama3.1-70B prefill phase with a 16K context window, based on LLMET simulation on a dual NVIDIA A100 GPU setup. On the 8x NVIDIA B200-like platform, extending the L2 cache from 128MB to 4GB saves the prefill energy by up to 24%. For the edge platform and workloads, the decode energy saving reaches 30% when increasing the 8MB cache size to 256MB. These results highlight the promise of ultra-large on-chip memories for energy-efficient LLM serving systems.
Innovation
Methods, ideas, or system contributions that make the work stand out.
M3D memory
cross-layer simulation
on-chip cache scaling
energy-efficient LLM serving
LLMET
🔎 Similar Papers
No similar papers found.
