Vllm Performance Tuning

vLLM performance tuning

Target: operators deploying models on new hardware, chasing throughput / latency / goodput SLOs, or diagnosing perf regressions. Current through v0.21.0 stable (2026-05-15); v0.20.x stable since 2026-04-27. Last freshened 2026-05-28.

Companion skills: vllm-benchmarking (measure), vllm-caching (KV), vllm-nvidia-hardware (GPU/GEMM), vllm-configuration (env vars), vllm-observability (metrics).

Tuning levers (apply by goal, not in fixed order)

Always first — characterize the workload. ISL / OSL / req/s / concurrency / SLO (P95 TTFT, P95 TPOT, P95 ITL). "Goodput" = tok/s/GPU under SLO, not raw tok/s. Everything below is keyed off these numbers.

Parallelism + MoE kernels (biggest single wins):

• Pick parallelism (see references/moe-and-ep.md) — model-fits-1-GPU → TP=1 + replicas (DP); MoE MLA (DeepSeek/Kimi-K2) → DP-attn + EP; multi-node → TP intra + PP inter OR Wide-EP.
• MoE on a new SKU → run benchmark_moe.py --tune — generates E=*,N=*,device_name=*.json configs. Without tuned configs vLLM logs "Using default MoE config. Performance might be sub-optimal!" = 20-40% throughput loss.
• Wide-EP (--enable-expert-parallel --enable-eplb --enable-dbo) for DeepSeek/Qwen3/Kimi-K2 at ≥16 GPUs.

Throughput / batching:

• auto_tune.sh (benchmarks/auto_tune/) sweeps max_num_seqs × max_num_batched_tokens.
• --gpu-memory-utilization — raise from 0.90 toward 0.95 until steady OOM margin, then back off. MoE: cap at 0.85 (all-to-all buffers not in accounting).
• Chunked prefill (always on in V1) — raise --max-num-batched-tokens (default 2048 since PR #10544) if TTFT > SLO; lower if ITL > SLO.

Latency / graph + compile:

• CUDA graphs — keep FULL_AND_PIECEWISE (default); align --cuda-graph-sizes with max_num_seqs*2.
• --async-scheduling — default-on in recent releases unless using spec-dec / PP / unsupported MM path.
• Compile cache — pre-bake $VLLM_CACHE_ROOT/torch_compile_cache on a representative pod; mount as PVC / bake into OCI layer.

Distributed (last resort, only when the lever above is exhausted):

• NCCL — on well-configured clouds do nothing. Bare-metal IB: NCCL_IB_HCA, NCCL_IB_GID_INDEX, NCCL_NET_GDR_LEVEL. Never NCCL_CUMEM_ENABLE=0 on GB200.
• PD disagg — reach for only after the above are exhausted and prefill interference is the actual bottleneck.

Triage tree ("why is it slow")

From Red Hat's 5-step triage (2026-03-09):

| Symptom | Look at | Common cause | |---|---|---| | TTFT high, queue empty | compute-bound prefill | chunked-prefill budget too low, no prefix cache, bad parallelism | | TTFT high, queue growing | capacity | raise replicas, raise max_num_seqs, check preemption rate | | TPOT high, TTFT fine | decode-bound | MoE kernel not tuned, wrong attention backend, async sched off | | ITL spikes | CUDA-graph miss | batch sizes fall outside captured buckets | | Preemptions climbing | KV thrashing | raise --swap-space, lower --max-num-seqs, or add replicas | | num_running --tp-size --enable-expert-parallel --tune --save-dir ./configs, then export VLLMTUNEDCONFIG_FOLDER=./configs`. Expected: 20-40% throughput recovery.

1. gpu_memory_utilization=0.95 on MoE. All-to-all staging buffers (DeepEP, NVSHMEM) aren't in the memory accountant. OOM at high concurrency. Fix: cap at 0.85 for MoE, 0.92 for dense.

1. TP not divisible by head count. Model has 32 heads, TP=7 → shape mismatch. Rule: num_heads % TP == 0 AND hidden_size % TP == 0.

1. DeepSeek-V3.2 at TP=8 on H100/H200/B200/B300. FlashMLA-Sparse only uses 16 heads per rank, padded to 64 → overhead. Fix: DP=8, EP=8, TP=1. (DeepSeek-V3.2 recipe)

1. Llama-4-Maverick with --enable-expert-parallel. Activation density 0.78% (1/128) — AllToAll overhead exceeds parallelism win. EP hurts 7-12% vs TP-only for Maverick. DeepSeek-R1 (3.13%) and Qwen3-235B (6.25%) benefit from EP. Rule: only enable EP when (experts_per_token / total_experts) > 2%.

1. MLA model with TP=8. Single KV head, TP duplicates ~84.5 GB KV cache per rank. Fix: DP-attention + EP-MoE splits KV to ~0.125 GB/GPU/request.

1. --async-scheduling with unsupported path. Structured outputs (fixed #26866), spec-dec (#24799 fixed), PP/struct-out/spec-dec/MM umbrella tracker #27679 closed 2025-12-29 — all sub-PRs merged. The vllm-ascend v0.11.0rc2 precision bug (#4649) is fixed — closed 2026-03-13; upgrade rather than disabling async-sched. Symptom: stall, latency regression, or precision loss. Fix: upgrade; only disable if you've reproduced the issue on your version.

1. Shipping VLLM_MOE_USE_DEEP_GEMM=1 blindly on H200. Between d83f3f7 and 5a84b76 the DeepGEMM MoE M 2% (below: TP wins) |

Source policy

All claims cite file:line, release-note PR refs, or issue IDs. Full anchor list + vendor-specific sources in references/sources.md. Compiled 2026-04-18 against v0.19.0; freshened 2026-05-28 against v0.21.0 stable (#28882 closed 2026-04-21, #27679 closed 2025-12-29, #31679 closed 2026-01-07, #34641 closed 2026-05-28, ascend #4649 closed 2026-03-13). Next refresh when v0.22.x stable ships or when remaining key regressions (#35048, #31475) close.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: air-gapped
• Source: air-gapped/skills
• License: MIT

Install and usage instructions live in the source repository linked above.