Vllm Bench Serve

vLLM Bench Serve

Benchmark vLLM or any OpenAI-compatible serving endpoint using the vllm bench serve CLI. Measures throughput, latency (TTFT, TPOT), and goodput against configurable request load.

Reference: vLLM Bench Serve Documentation

Prerequisites

• vLLM installed (or any OpenAI-compatible server running)
• A vLLM server or API endpoint already serving a model
• Python environment with vLLM for the benchmark client

Quick Start

Basic benchmark against local vLLM server (default random dataset, 1000 prompts):

vllm bench serve \
  --backend openai-chat \
  --host 127.0.0.1 \
  --port 8000 \
  --model Qwen/Qwen2.5-1.5B-Instruct \
  --endpoint /v1/chat/completions

Save results to JSON:

vllm bench serve \
  --backend openai-chat \
  --host 127.0.0.1 \
  --port 8000 \
  --model Qwen/Qwen2.5-1.5B-Instruct \
  --endpoint /v1/chat/completions \
  --save-result \
  --result-dir ./bench-results \
  --metadata "version=0.6.0" "tp=1"

> Note: When using --backend openai-chat, you must specify --endpoint /v1/chat/completions (default is /v1/completions).

Core Arguments

| Argument | Default | Description | |----------|---------|-------------| | --backend | openai | Backend type: openai, openai-chat, openai-embeddings, vllm, vllm-pooling, vllm-rerank, etc. | | --host | 127.0.0.1 | Server host | | --port | 8000 | Server port | | --base-url | - | Alternative: full base URL instead of host:port | | --endpoint | /v1/completions | API endpoint; use /v1/chat/completions for openai-chat | | --model | (from /v1/models) | Model name | | --num-prompts | 1000 | Number of prompts to process | | --request-rate | inf | Requests per second; inf = burst all at once | | --max-concurrency | - | Max concurrent requests (caps parallelism) | | --num-warmups | 0 | Warmup requests before measuring |

Datasets

| --dataset-name | Use Case | |------------------|----------| | random | Synthetic random prompts (default) | | sharegpt | ShareGPT conversation format; requires --dataset-path | | sonnet | Sonnet-style prompts | | hf | HuggingFace dataset; requires --dataset-path (dataset ID) | | custom / custom_mm | Custom dataset; requires --dataset-path | | prefix_repetition | Prefix repetition benchmark | | random-mm | Random multimodal (images/videos) | | spec_bench | Spec bench dataset |

Dataset-specific options (examples):

# Random: control input/output length
--dataset-name random --random-input-len 1024 --random-output-len 128

# Sonnet defaults: input 550, output 150, prefix 200
--dataset-name sonnet --sonnet-input-len 550 --sonnet-output-len 150

# HuggingFace dataset
--dataset-name hf --dataset-path "lmarena-ai/VisionArena-Chat" --hf-split test

# General overrides (map to dataset-specific args)
--input-len 512 --output-len 256

Load Control

# Fixed request rate (Poisson process)
--request-rate 10

# More bursty arrivals (gamma distribution, burstiness  vllm bench serve \
  --backend openai-chat --host 127.0.0.1 --port 8000 \
  --model Qwen/Qwen2.5-1.5B-Instruct \
  --endpoint /v1/chat/completions \
  --dataset-name random --num-prompts 100

Troubleshooting

• Connection refused: Ensure the server is running and --host/--port or --base-url are correct.
• Model not found: Pass --model explicitly or ensure /v1/models returns the model.
• URL must end with chat/completions: Use --endpoint /v1/chat/completions when --backend openai-chat.
• Rate limit / 429: Reduce --request-rate or --max-concurrency.
• Ready check: Use --ready-check-timeout-sec 60 to wait for the endpoint before benchmarking.
• SSL: Use --insecure for self-signed certificates.

Notes

• For embeddings/rerank benchmarks, use --backend openai-embeddings, vllm-pooling, or vllm-rerank.
• --profile requires --profiler-config on the server for vLLM profiling.
• Goodput SLOs are useful for SLA-style analysis; see DistServe paper for details.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: vllm-project
• Source: vllm-project/vllm-skills
• License: Apache-2.0

Install and usage instructions live in the source repository linked above.