New MLSys 2026 paper in collaboration with Paderborn University, Germany
In collaboration with Prof. Lin Wang and his PhD student, Huzaifa Shaaban Kabakibo, we are excited to share that our paper on analyzing the cold-start process of vLLM has been accepted to MLSys 2026!
vLLM is a popular inference serving engine, and as demand for high-performance LLM inference continues to grow, flexible resource and infrastructure provisioning is becoming increasingly important. To enable this, it is essential to understand how quickly a new vLLM inference engine can be provisioned.
In this work, we take a deep dive into vLLM’s startup routine, breaking down its core steps and dependencies across 20+ LLM models and multiple GPU and CPU architectures. Our study provides a detailed characterization of vLLM’s cold-start behavior and sheds light on the factors that determine how quickly inference capacity can be brought online.
Title: Breaking the Ice: Analyzing Cold Start Latency in vLLM
Authors: Huzaifa Shaaban Kabakibo (Paderborn University, Germany), Animesh Trivedi (IBM Research Zurich), Lin Wang (Paderborn University, Germany)
Abstract: As scalable inference services become popular, the cold start latency of an inference engine becomes important. Today, vLLM has evolved into the de-facto inference engine of choice for many inference workloads. Although popular, due to its complexity and rapid evolution, there has not been a systematic study on the startup latency of its engine. With major architectural innovations under it (e.g., the V1 API, introduction of this http URL), in this paper, we present the first detailed performance characterization of vLLM startup latency. We break down the startup process into six foundational steps and demonstrate that this process is predominantly CPU-bound. Each step exhibits consistent and interpretable scaling trends with respect to model- and system-level parameters, enabling fine-grained attribution of latency sources. Building on these insights, we develop a lightweight analytical model that accurately predicts vLLM’s startup latency for a given hardware configuration, providing actionable guidance for resource planning in large-scale inference environments. All our benchmarking datasets, analysis tools, and prediction scripts are open-sourced at https://github.com/upb-cn/vllm-startup-profiler.
OpenReview: https://openreview.net/forum?id=eoEobeKTNZ