🤖 AI Summary
This work addresses the cold-start latency challenges faced by vLLM in large-scale inference serving, a problem whose root causes have not been systematically investigated. The study presents the first fine-grained decomposition of vLLM’s cold-start process, identifying six critical stages and revealing their CPU-bound nature. Building upon this insight, the authors develop an interpretable latency attribution and prediction model that integrates both model-level and system-level parameters. By leveraging performance profiling, parametric modeling, and optimizations such as torch.compile, the proposed approach achieves high-accuracy prediction of startup latency across diverse hardware configurations. The project publicly releases the complete suite of analysis tools and datasets, offering practical foundations for resource provisioning and performance optimization in inference serving systems.
📝 Abstract
As scalable inference services become popular, the cold start latency of an inference engine becomes important. Today, vLLM has evolved into the de facto inference engine of choice for many inference workloads. Although popular, due to its complexity and rapid evolution, there has not been a systematic study of its startup latency. With major architectural innovations such as the V1 API and the introduction of torch.compile, this paper presents the first detailed performance characterization of vLLM startup latency. We break down the startup process into six foundational steps and demonstrate that it is predominantly CPU bound. Each step exhibits consistent and interpretable scaling trends with respect to model-level and system-level parameters, enabling fine-grained attribution of latency sources. Building on these insights, we develop a lightweight analytical model that accurately predicts vLLM startup latency for a given hardware configuration, providing actionable guidance for resource planning in large-scale inference environments. All benchmarking datasets, analysis tools, and prediction scripts are open sourced at https://github.com/upb-cn/vllm-startup-profiler.