Vllm Prefix Cache Bench

vLLM Prefix Caching Benchmark

Benchmark the efficiency of vLLM's automatic prefix caching (APC) feature. The offline script benchmarks/benchmark_prefix_caching.py runs directly against the vLLM engine (no server required). For online/serving tests, use vllm bench serve with the prefix_repetition dataset.

When to use

• User wants to measure the performance impact of prefix caching for repeated or partially-shared prompts.
• User wants to compare throughput/latency with and without --enable-prefix-caching.
• User wants to test prefix caching using a fixed synthetic prompt, a real dataset (e.g. ShareGPT), or a synthetic prefix/suffix repetition pattern.

Option 1 (default). Fixed Prompt with Prefix Caching

Runs a synthetic benchmark with a fixed prompt repeated multiple times to directly measure cache hit efficiency. No dataset download required.

python3 benchmarks/benchmark_prefix_caching.py \
  --model Qwen/Qwen3-8B \
  --enable-prefix-caching \
  --num-prompts 1 \
  --repeat-count 100 \
  --input-length-range 128:256

To compare against the baseline without caching:

python3 benchmarks/benchmark_prefix_caching.py \
  --model Qwen/Qwen3-8B \
  --no-enable-prefix-caching \
  --num-prompts 1 \
  --repeat-count 100 \
  --input-length-range 128:256

Option 2. ShareGPT Dataset with Prefix Caching

Uses real-world conversational data from ShareGPT to evaluate prefix caching with naturally occurring prompt sharing.

First, download the dataset:

wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json

Then run the benchmark:

python3 benchmarks/benchmark_prefix_caching.py \
  --model Qwen/Qwen3-8B \
  --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json \
  --enable-prefix-caching \
  --num-prompts 20 \
  --repeat-count 5 \
  --input-length-range 128:256

Option 3. Prefix Repetition Dataset (Online)

Uses vllm bench serve with the synthetic prefix_repetition dataset to test caching via the serving API. This requires a running vLLM server.

First, start the server:

vllm serve Qwen/Qwen3-8B

Then run the benchmark:

vllm bench serve \
  --backend openai \
  --model Qwen/Qwen3-8B \
  --dataset-name prefix_repetition \
  --num-prompts 100 \
  --prefix-repetition-prefix-len 512 \
  --prefix-repetition-suffix-len 128 \
  --prefix-repetition-num-prefixes 5 \
  --prefix-repetition-output-len 128

Key parameters for prefix_repetition:

| Parameter | Description | |---|---| | --prefix-repetition-prefix-len | Number of tokens in the shared prefix portion | | --prefix-repetition-suffix-len | Number of tokens in the unique suffix portion | | --prefix-repetition-num-prefixes | Number of distinct prefixes to cycle through | | --prefix-repetition-output-len | Number of output tokens to generate per request |

Notes

• Run all commands from the root of the vLLM repository (cd vllm).
• Keep the default model (Qwen/Qwen3-8B) unless the user specifies a different one or the model is unavailable; change only --model.
• --repeat-count in Option 1 and 2 controls how many times each sampled prompt is replayed; higher values increase cache hit rate.
• --input-length-range accepts a min:max token range, e.g. 128:256.
• For multi-GPU setups, add --tensor-parallel-size .
• To test different hash algorithms for prefix caching internals, use --prefix-caching-hash-algo xxhash (requires pip install xxhash).

Arguments for benchmark_prefix_caching.py

| Argument | Required | Description | |---|---|---| | --model | Yes | Model name or path (HuggingFace ID or local path) | | --num-prompts | Yes | Number of prompts to process | | --input-length-range | Yes | Token length range for inputs, e.g. 128:256 | | --repeat-count | No | Number of times each prompt is repeated (default: 1) | | --dataset-path | No | Path to a dataset file (e.g. ShareGPT JSON). Omit for synthetic fixed-prompt mode | | --prefix-len | No | Fixed prefix token length to prepend to every prompt | | --output-len | No | Number of output tokens to generate per request | | --sort | No | Sort prompts by length before benchmarking | | --enable-prefix-caching / --no-enable-prefix-caching | No | Toggle APC (recommended: enable to test caching) | | --prefix-caching-hash-algo | No | Hash algorithm: sha256, sha256_cbor, xxhash, xxhash_cbor | | --tensor-parallel-size | No | Number of GPUs for tensor parallelism | | --disable-detokenize | No | Skip detokenization to reduce overhead |

Troubleshooting

• If python3 benchmarks/*.py reports file not found, locate your local vLLM repository first and run the command from that repo root.
• If you do not have the repository yet, clone it and continue:

git clone https://github.com/vllm-project/vllm
cd vllm

• If HuggingFace model download fails due to access restrictions, set your token: export HF_TOKEN= or pass --hf-token .
• If xxhash or cbor2 is not installed and you use those hash algorithms, install them first: pip install xxhash cbor2.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: vllm-project
• Source: vllm-project/vllm-skills
• License: Apache-2.0

Install and usage instructions live in the source repository linked above.