🤖 AI Summary
This work addresses a critical flaw in existing LLM inference benchmarks, where single-process clients under high concurrency suffer from Python’s Global Interpreter Lock (GIL), leading to severe distortion in Time-to-First-Token (TTFT) and Time Per Output Token (TPOT) metrics. The study is the first to model the client as an M/G/1 queue, uncovering systematic bias introduced by queuing effects. To rectify this, the authors propose a multi-process, unbiased evaluation framework accompanied by a normalized metric—Normalized Time Per Output Token (NTPOT). This approach effectively eliminates client-side bottlenecks, enabling accurate and reproducible performance evaluation at scale, supporting thousands of queries per second. It substantially reduces the latency overestimation—often several-fold—in conventional benchmarks, thereby reflecting the true performance of production-grade LLM services.
📝 Abstract
As Large Language Models (LLMs) transition from research environments to production deployments, evaluating their performance against strict Service Level Objectives (SLOs) has become critical. However, current evaluation methodologies suffer from severe measurement bias at scale. We demonstrate that widely used benchmarking utilities rely on single-process, asyncio-driven architectures that introduce fundamental client-side queuing bottlenecks under high concurrency. By modeling the benchmarking client as an $M/G/1$ queue, we mathematically demonstrate how the Python Global Interpreter Lock (GIL) artificially inflates Time to First Token (TTFT) and Time Per Output Token (TPOT) metrics as request rates scale. To resolve this systematic inaccuracy, we propose an unbiased, multi-process evaluation framework that effectively distributes client-side load, ensuring negligible queuing overhead. Furthermore, we formalize a composite metric, Normalized Time Per Output Token (NTPOT), to robustly amortize end-to-end latency, including prefill and scheduling delays across sequence lengths. Our empirical evaluation demonstrates that this methodology successfully isolates pure serving engine performance, enabling accurate, reproducible profiling of LLMs at production scales exceeding thousands of queries per second.