# Vllm Prefix Cache Bench

> This is a skill for benchmarking the efficiency of automatic prefix caching in vLLM using fixed prompts, real-world datasets, or synthetic prefix/suffix patterns. Use when the user asks to benchmark prefix caching hit rate, caching efficiency, or repeated-prompt performance in vLLM.

- **Type:** Skill
- **Install:** `agentstack add skill-vllm-project-vllm-skills-vllm-prefix-cache-bench`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [vllm-project](https://agentstack.voostack.com/s/vllm-project)
- **Installs:** 0
- **Category:** [Agent Skills](https://agentstack.voostack.com/c/agent-skills)
- **Latest version:** 0.1.0
- **License:** Apache-2.0
- **Upstream author:** [vllm-project](https://github.com/vllm-project)
- **Source:** https://github.com/vllm-project/vllm-skills/tree/main/plugins/vllm-skills/skills/vllm-prefix-cache-bench

## Install

```sh
agentstack add skill-vllm-project-vllm-skills-vllm-prefix-cache-bench
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# vLLM Prefix Caching Benchmark

Benchmark the efficiency of vLLM's automatic prefix caching (APC) feature. The offline script `benchmarks/benchmark_prefix_caching.py` runs directly against the vLLM engine (no server required). For online/serving tests, use `vllm bench serve` with the `prefix_repetition` dataset.

## When to use

- User wants to measure the performance impact of prefix caching for repeated or partially-shared prompts.
- User wants to compare throughput/latency with and without `--enable-prefix-caching`.
- User wants to test prefix caching using a fixed synthetic prompt, a real dataset (e.g. ShareGPT), or a synthetic prefix/suffix repetition pattern.

## Option 1 (default). Fixed Prompt with Prefix Caching

Runs a synthetic benchmark with a fixed prompt repeated multiple times to directly measure cache hit efficiency. No dataset download required.

```bash
python3 benchmarks/benchmark_prefix_caching.py \
  --model Qwen/Qwen3-8B \
  --enable-prefix-caching \
  --num-prompts 1 \
  --repeat-count 100 \
  --input-length-range 128:256
```

To compare against the baseline without caching:

```bash
python3 benchmarks/benchmark_prefix_caching.py \
  --model Qwen/Qwen3-8B \
  --no-enable-prefix-caching \
  --num-prompts 1 \
  --repeat-count 100 \
  --input-length-range 128:256
```

## Option 2. ShareGPT Dataset with Prefix Caching

Uses real-world conversational data from ShareGPT to evaluate prefix caching with naturally occurring prompt sharing.

First, download the dataset:

```bash
wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json
```

Then run the benchmark:

```bash
python3 benchmarks/benchmark_prefix_caching.py \
  --model Qwen/Qwen3-8B \
  --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json \
  --enable-prefix-caching \
  --num-prompts 20 \
  --repeat-count 5 \
  --input-length-range 128:256
```

## Option 3. Prefix Repetition Dataset (Online)

Uses `vllm bench serve` with the synthetic `prefix_repetition` dataset to test caching via the serving API. This requires a running vLLM server.

First, start the server:

```bash
vllm serve Qwen/Qwen3-8B
```

Then run the benchmark:

```bash
vllm bench serve \
  --backend openai \
  --model Qwen/Qwen3-8B \
  --dataset-name prefix_repetition \
  --num-prompts 100 \
  --prefix-repetition-prefix-len 512 \
  --prefix-repetition-suffix-len 128 \
  --prefix-repetition-num-prefixes 5 \
  --prefix-repetition-output-len 128
```

Key parameters for `prefix_repetition`:

| Parameter | Description |
|---|---|
| `--prefix-repetition-prefix-len` | Number of tokens in the shared prefix portion |
| `--prefix-repetition-suffix-len` | Number of tokens in the unique suffix portion |
| `--prefix-repetition-num-prefixes` | Number of distinct prefixes to cycle through |
| `--prefix-repetition-output-len` | Number of output tokens to generate per request |

## Notes

- Run all commands from the root of the vLLM repository (`cd vllm`).
- Keep the default model (`Qwen/Qwen3-8B`) unless the user specifies a different one or the model is unavailable; change only `--model`.
- `--repeat-count` in Option 1 and 2 controls how many times each sampled prompt is replayed; higher values increase cache hit rate.
- `--input-length-range` accepts a `min:max` token range, e.g. `128:256`.
- For multi-GPU setups, add `--tensor-parallel-size `.
- To test different hash algorithms for prefix caching internals, use `--prefix-caching-hash-algo xxhash` (requires `pip install xxhash`).

## Arguments for `benchmark_prefix_caching.py`

| Argument | Required | Description |
|---|---|---|
| `--model` | Yes | Model name or path (HuggingFace ID or local path) |
| `--num-prompts` | Yes | Number of prompts to process |
| `--input-length-range` | Yes | Token length range for inputs, e.g. `128:256` |
| `--repeat-count` | No | Number of times each prompt is repeated (default: 1) |
| `--dataset-path` | No | Path to a dataset file (e.g. ShareGPT JSON). Omit for synthetic fixed-prompt mode |
| `--prefix-len` | No | Fixed prefix token length to prepend to every prompt |
| `--output-len` | No | Number of output tokens to generate per request |
| `--sort` | No | Sort prompts by length before benchmarking |
| `--enable-prefix-caching` / `--no-enable-prefix-caching` | No | Toggle APC (recommended: enable to test caching) |
| `--prefix-caching-hash-algo` | No | Hash algorithm: `sha256`, `sha256_cbor`, `xxhash`, `xxhash_cbor` |
| `--tensor-parallel-size` | No | Number of GPUs for tensor parallelism |
| `--disable-detokenize` | No | Skip detokenization to reduce overhead |

## Troubleshooting

- If `python3 benchmarks/*.py` reports file not found, locate your local vLLM repository first and run the command from that repo root.
- If you do not have the repository yet, clone it and continue:

```bash
git clone https://github.com/vllm-project/vllm
cd vllm
```

- If HuggingFace model download fails due to access restrictions, set your token: `export HF_TOKEN=` or pass `--hf-token `.
- If `xxhash` or `cbor2` is not installed and you use those hash algorithms, install them first: `pip install xxhash cbor2`.

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [vllm-project](https://github.com/vllm-project)
- **Source:** [vllm-project/vllm-skills](https://github.com/vllm-project/vllm-skills)
- **License:** Apache-2.0

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** yes
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-vllm-project-vllm-skills-vllm-prefix-cache-bench
- Seller: https://agentstack.voostack.com/s/vllm-project
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.