SLIM: Saturation-Aware Lightweight Performance Modeling for LLM Serving

2026-07-31

AI Summary

In large language model serving, increasing batch sizes often leads to throughput saturation, higher latency, and inefficient GPU memory utilization. Through GPU hardware profiling, this study identifies that during the decoding phase, the attention kernel becomes bottlenecked by DRAM bandwidth due to growing context lengths, with its constant arithmetic intensity being the root cause of throughput limitations. To address this, the authors propose SLIM, a semi-analytical performance model that integrates Transformer computation and memory traffic modeling to accurately predict throughput and latency. They further design BCA, a saturation-aware batching configuration advisor, to guide efficient batching strategies. Experiments demonstrate SLIM’s strong generalization across diverse scenarios, outperforming existing baselines while reducing GPU memory usage by up to 55 GB for OPT models with negligible throughput loss.

Abstract

Large language model (LLM) serving commonly increases batch size to improve throughput, but performance eventually reaches a deployment-dependent plateau beyond which larger batches provide marginal gains while increasing latency and GPU memory consumption. Previous studies have attributed this behavior to HBM/DRAM bandwidth limitations, but the underlying causes have primarily been supported by conceptual arguments or high-level performance observations. As our first contribution, we present a detailed GPU characterization using hardware profiling techniques, demonstrating that throughput saturation originates in the attention kernels during the decode phase. Specifically, we show that their nearly constant arithmetic intensity as active-context lengths increases -not merely larger batch sizes- drives DRAM-bandwidth saturation, while the achieved compute throughput remains far below the hardware limit. Building on this analysis, we present the Batching Configuration Advisor (BCA), which selects the highest-throughput batching configuration satisfying a target latency constraint and identifies up to 55 GB of GPU memory allocation that can be avoided for the evaluated OPT models with minimal throughput loss. To enable these recommendations, we introduce SLIM (Saturation-Aware Lightweight Performance Model), a semi-analytical model that predicts LLM inference throughput and latency from analytical formulations of Transformer computation and memory traffic. Across the evaluated scenarios, SLIM outperforms representative performance-modeling baselines while successfully generalizing to previously unseen operating conditions.

Problem

• LLM serving
• throughput saturation
• batching
• DRAM bandwidth
• performance modeling

Innovation

• SLIM
• throughput saturation
• attention kernel
• batching configuration
• performance modeling