As a part of my first vLLM contribution, I implemented support for replaying growing number of inference traces in vllm benchmarking framework.
PR: https://github.com/vllm-project/vllm/pull/39795
With this PR, one can replay timed traces in vLLM to enable realistic benchmarking with production-style traces from workloads such as MoonshotAI and Alibaba. It introduces a new timed_trace dataset that reconstructs prompts from input/output lengths and hashed prompt IDs, preserving caching behavior while replaying requests according to their original timing. This is particularly useful when benchmarking KVCache behavior at scale. This makes it possible to evaluate vLLM under realistic, time-varying workloads—using real traces rather than synthetic request rates.
How to use this feature in benchmarking
# download traces
$ weget https://github.com/kvcache-ai/Mooncake/blob/main/FAST25-release/traces/conversation_trace.jsonl
# start the server on the one shell
$ vllm serve Qwen/Qwen3.5-2B --model-impl vllm --block-size 16
on another shell
# start the benchmarking client
vllm bench serve --model Qwen/Qwen3.5-2B --dataset-name=timed_trace \
--num-prompts 50 --host 127.0.0.1 --port 8000 \
--dataset-path conversation_trace.jsonl --ignore-eos \
--self-timed --timed-trace-label-timestamp timestamp \
--timed-trace-chunk-hash-size 512 \
--timed-trace-label-hash-ids hash_ids \
--timed-trace-label-input-length input_length \
--timed-trace-label-output-length output_length \
--timed-trace-sec-multiplier 0.001
You will get output like:
============ Serving Benchmark Result ============
Successful requests: 50
Failed requests: 0
Benchmark duration (s): 28.35
Total input tokens: 601420
Total generated tokens: 18175
Request throughput (req/s): 1.76
Output token throughput (tok/s): 641.16
Peak output token throughput (tok/s): 1987.00
Peak concurrent requests: 42.00
Total token throughput (tok/s): 21857.47
---------------Time to First Token----------------
Mean TTFT (ms): 3870.54
Median TTFT (ms): 4202.70
P99 TTFT (ms): 8797.25
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 36.17
Median TPOT (ms): 35.55
P99 TPOT (ms): 72.00
---------------Inter-token Latency----------------
Mean ITL (ms): 31.66
Median ITL (ms): 18.01
P99 ITL (ms): 93.68
==================================================