InstantInfer: Enabling Fast LLM Cold Start with Communicating Finite Automata

2026-07-21

AI Summary

This work addresses the inefficiency of cold-start in large language model (LLM) inference services, which stems from serial initialization and numerous fine-grained I/O requests. The authors propose a Communicating Finite Automaton (CFA) abstraction to systematically identify cross-component optimization opportunities and design a programming framework that enables safe concurrent execution of complex, heterogeneous components while preserving original program structure. This is the first application of CFA to LLM cold-start optimization. By restructuring key paths in vLLM—such as process tree creation, tensor loading, and model switching—and integrating I/O coalescing with concurrency control, the approach achieves correctness and performance without code rewriting. Experiments demonstrate up to 7.2× cold-start acceleration across diverse GPUs, workloads, and model scales, significantly improving LLM service responsiveness and robustness.

Abstract

Cold starts in large language model (LLM) inference services significantly affect user experience, yet they remain inefficient due to sequential initialization and a massive number of fine-grained I/O requests issued by complex software components. Although refactoring the program can yield advantages such as concurrent execution and I/O merging, this approach is error-prone and carries correctness risks when dealing with massive, heterogeneous components. We propose the Communicating Finite Automata (CFA) abstraction to systematically analyze cross-component optimization opportunities, and design a programming framework to enable CFA-based component program refactoring. This framework preserves the original sequential program structure while enabling safe concurrent component execution. We prove the correctness of the program refactoring. We apply the CFA abstraction and framework to refactor process tree creation, tensor loading, and model switching in vLLM, forming a new cold-start system named InstantInfer. Extensive experiments demonstrate that InstantInfer substantially accelerates LLM cold starts (achieving up to 7.2 times speedup) and exhibits robustness across diverse GPUs, workloads, and scales.

Problem

• cold start
• large language model
• I/O requests
• initialization efficiency
• inference latency

Innovation

• Communicating Finite Automata
• LLM cold start
• concurrent execution
• program refactoring
• vLLM