Vllm Benchmark Result Analysis

vLLM Benchmark Summary

Workflow

1. Identify the input — the user pastes combined text containing a "before" block and an "after" block (or provides two separate files).
2. Save the input to /tmp/bench_input.txt, then run the script.
3. Run the script using the bundled ./scripts/compare_benchmarks.py. This generates the Performance Metrics table.
4. Generate the Summary — after the script runs, analyze the table data and write a narrative ## AI Summary section (see guidelines below).
5. Assemble and save — append the Summary to the output file, then show the complete report to the user.

Running the Script

Combined text input (most common)

Save the user's pasted text to /tmp/bench_input.txt, then:

python3 /Users/shanshan-shen/.claude/skills/vllm-benchmark-result-analysis/scripts/compare_benchmarks.py \
    /tmp/bench_input.txt \
    --output-dir /Users/shanshan-shen/.claude/skills/vllm-benchmark-result-analysis/outputs \
    --title "vLLM Benchmark Comparison"

Two separate files

python3 /Users/shanshan-shen/.claude/skills/vllm-benchmark-result-analysis/scripts/compare_benchmarks.py \
    --before before.txt --after after.txt \
    --output-dir /Users/shanshan-shen/.claude/skills/vllm-benchmark-result-analysis/outputs \
    --title "vLLM Benchmark Comparison"

Input Format

The script expects the standard vLLM benchmark output block:

Before this PR:          ← any line containing "before" (case-insensitive)

============ Serving Benchmark Result ============
Metric Name:                     value
...
==================================================

After this PR:           ← any line containing "after" (case-insensitive)

============ Serving Benchmark Result ============
...
==================================================

Metric lines must follow the pattern Metric Name: . Section separator lines (===, ---) are ignored automatically.

Output

The script creates a report file under /Users/shanshan-shen/.claude/skills/vllm-benchmark-result-analysis/outputs/ containing:

• Performance Metrics table — metrics grouped into Throughput and Latency sections, with left-aligned columns and percentage comparison (e.g. +0.54% ↑, -9.86% ↓)

After the script runs, read the output file path from the script's stdout, then append a ## AI Summary section with narrative bullet points, and save the complete report.

Comparison column format

• Two decimal places with sign and direction arrow: +0.54% ↑, -9.86% ↓
• Arrow indicates direction of change only (↑ = value increased, ↓ = value decreased), not whether it is an improvement

Summary Writing Guidelines

Write 4–6 narrative bullet points covering:

• The standout improvement (largest positive delta for a "lower is better" metric, or largest increase for "higher is better")
• Notable TTFT / TPOT / ITL changes with before→after values
• Overall throughput trend (flat, improved, or regressed)
• Any minor regressions worth flagging, noting if they may be within noise
• Any peak or burst metric anomalies with context

Direction conventions — use these to frame improvements vs. regressions:

• Lower is better: TTFT, TPOT, ITL (all variants), Benchmark duration, Failed requests
• Higher is better: Request throughput, Output/Total token throughput

Style rules:

• Use **bold** to emphasize key percentages and before→after values (e.g. **-18.45%**, 439ms → 359ms)
• Do not use ✅/❌ markers — write in plain narrative style
• Keep bullets concise (1–2 sentences each)

Table format

Refer to the reference/reference.md file for a full example of the generated report format and content.

Notes

• Always use --title "vLLM Benchmark Comparison" unless the user specifies a different title.
• Save the output file into /Users/shanshan-shen/.claude/skills/vllm-benchmark-result-analysis/outputs.
• The script requires only the Python standard library — no extra dependencies.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: shen-shanshan
• Source: shen-shanshan/vllm-dev-skills
• License: Apache-2.0
• Homepage: https://zhuanlan.zhihu.com/p/2031696581678866733

Install and usage instructions live in the source repository linked above.