Problem
Research questions and friction points this paper is trying to address.
heterogeneous NPUs
LLM serving
energy efficiency
cost efficiency
auto-scaling
Auto-Scaling Heterogeneous Neural Processing Units for Energy and Cost-Efficient LLM Serving
🤖 AI Summary
This work addresses the low utilization of heterogeneous neural processing unit (NPU) resources in cloud platforms and the insufficient energy and cost efficiency of large language model (LLM) serving. To tackle these challenges, the authors propose NeuScale, an automated scaling framework that introduces vPod—a novel hardware abstraction for unified management of multi-generation NPUs. NeuScale integrates a lightweight roofline performance model to dynamically allocate optimal resource configurations for incoming LLM inference requests and enables fine-grained, two-dimensional elasticity across both per-instance configuration and the number of instances. Experimental evaluation demonstrates that NeuScale substantially improves cost efficiency and SLO compliance for LLM services, with its effectiveness validated in real-world heterogeneous NPU environments.
📝 Abstract
To meet the ever-increasing computing demands of large language model (LLM) services, modern cloud platforms have widely deployed neural processing units (NPUs). These NPU chips have been developed and evolved at an incredibly fast pace, this inevitably produces heterogeneous compute pools backed by different versions of NPU chips. Unfortunately, due to the lack of system and architecture support for managing NPU heterogeneity in the cloud, it is unclear how to best utilize heterogeneous NPUs to maximize the energy and cost efficiency for LLM services.
In this paper, we first conduct a characterization study of various generations of real NPU chips to demonstrate the potential benefits on energy/cost efficiency and performance by utilizing heterogeneous NPU chips. To realize these benefits, we present NeuScale, an auto-scaling framework to automatically exploit heterogeneous NPUs for cloud platforms. NeuScale manages heterogeneous NPU resources with a new vPod abstraction, which abstracts the core hardware parameters of different NPU versions and provides compatibility with existing ML frameworks. It makes the best-fit vPod allocations for different LLM inference requests using an intuitive and lightweight roofline-based analysis. It supports fine-grained dynamic NPU resource provisioning by adjusting both the vPod configuration (i.e., scaling up/down) and the number of vPods (e.g., scaling in/out). To validate the benefits of NeuScale at scale, we implement it with a production-level NPU simulator. Our evaluation with popular LLMs shows that NeuScale can significantly improve cost efficiency and service-level objective (SLO) satisfaction rate by best utilizing heterogeneous NPU resources.
Innovation
Methods, ideas, or system contributions that make the work stand out.
heterogeneous NPUs
auto-scaling
vPod abstraction
LLM serving
energy efficiency
🔎 Similar Papers
