# Vllm Benchmark Result Analysis

> Compare vLLM serving benchmark outputs before and after a code change. Parses plain-text vLLM benchmark output (the "Serving Benchmark Result" block), computes per-metric percentage changes with improvement/regression markers, generates a Markdown report with a full metrics table and a key-changes summary, and saves it to ./outputs/. Use when the user pastes or provides vLLM benchmark output and…

- **Type:** Skill
- **Install:** `agentstack add skill-shen-shanshan-vllm-dev-skills-vllm-benchmark-result-analysis`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [shen-shanshan](https://agentstack.voostack.com/s/shen-shanshan)
- **Installs:** 0
- **Category:** [Data & Analytics](https://agentstack.voostack.com/c/data-and-analytics)
- **Latest version:** 0.1.0
- **License:** Apache-2.0
- **Upstream author:** [shen-shanshan](https://github.com/shen-shanshan)
- **Source:** https://github.com/shen-shanshan/vllm-dev-skills/tree/master/skills/vllm-benchmark-result-analysis
- **Website:** https://zhuanlan.zhihu.com/p/2031696581678866733

## Install

```sh
agentstack add skill-shen-shanshan-vllm-dev-skills-vllm-benchmark-result-analysis
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# vLLM Benchmark Summary

## Workflow

1. **Identify the input** — the user pastes combined text containing a "before" block and an "after" block (or provides two separate files).
2. **Save the input** to `/tmp/bench_input.txt`, then run the script.
3. **Run the script** using the bundled `./scripts/compare_benchmarks.py`. This generates the Performance Metrics table.
4. **Generate the Summary** — after the script runs, analyze the table data and write a narrative `## AI Summary` section (see guidelines below).
5. **Assemble and save** — append the Summary to the output file, then show the complete report to the user.

## Running the Script

### Combined text input (most common)

Save the user's pasted text to `/tmp/bench_input.txt`, then:

```bash
python3 /Users/shanshan-shen/.claude/skills/vllm-benchmark-result-analysis/scripts/compare_benchmarks.py \
    /tmp/bench_input.txt \
    --output-dir /Users/shanshan-shen/.claude/skills/vllm-benchmark-result-analysis/outputs \
    --title "vLLM Benchmark Comparison"
```

### Two separate files

```bash
python3 /Users/shanshan-shen/.claude/skills/vllm-benchmark-result-analysis/scripts/compare_benchmarks.py \
    --before before.txt --after after.txt \
    --output-dir /Users/shanshan-shen/.claude/skills/vllm-benchmark-result-analysis/outputs \
    --title "vLLM Benchmark Comparison"
```

## Input Format

The script expects the standard vLLM benchmark output block:

```
Before this PR:          ← any line containing "before" (case-insensitive)

============ Serving Benchmark Result ============
Metric Name:                     value
...
==================================================

After this PR:           ← any line containing "after" (case-insensitive)

============ Serving Benchmark Result ============
...
==================================================
```

Metric lines must follow the pattern `Metric Name: `. Section separator lines (`===`, `---`) are ignored automatically.

## Output

The script creates a report file under `/Users/shanshan-shen/.claude/skills/vllm-benchmark-result-analysis/outputs/` containing:

- **Performance Metrics table** — metrics grouped into **Throughput** and **Latency** sections, with left-aligned columns and percentage comparison (e.g. `+0.54% ↑`, `-9.86% ↓`)

After the script runs, **read the output file path from the script's stdout**, then **append a `## AI Summary` section** with narrative bullet points, and save the complete report.

### Comparison column format

- Two decimal places with sign and direction arrow: `+0.54% ↑`, `-9.86% ↓`
- Arrow indicates direction of change only (↑ = value increased, ↓ = value decreased), not whether it is an improvement

### Summary Writing Guidelines

Write 4–6 narrative bullet points covering:
- The standout improvement (largest positive delta for a "lower is better" metric, or largest increase for "higher is better")
- Notable TTFT / TPOT / ITL changes with before→after values
- Overall throughput trend (flat, improved, or regressed)
- Any minor regressions worth flagging, noting if they may be within noise
- Any peak or burst metric anomalies with context

Direction conventions — use these to frame improvements vs. regressions:
- **Lower is better**: TTFT, TPOT, ITL (all variants), Benchmark duration, Failed requests
- **Higher is better**: Request throughput, Output/Total token throughput

Style rules:
- Use `**bold**` to emphasize key percentages and before→after values (e.g. `**-18.45%**`, `439ms → 359ms`)
- Do **not** use ✅/❌ markers — write in plain narrative style
- Keep bullets concise (1–2 sentences each)

### Table format

Refer to the `reference/reference.md` file for a full example of the generated report format and content.

## Notes

- Always use `--title "vLLM Benchmark Comparison"` unless the user specifies a different title.
- Save the output file into `/Users/shanshan-shen/.claude/skills/vllm-benchmark-result-analysis/outputs`.
- The script requires only the Python standard library — no extra dependencies.

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [shen-shanshan](https://github.com/shen-shanshan)
- **Source:** [shen-shanshan/vllm-dev-skills](https://github.com/shen-shanshan/vllm-dev-skills)
- **License:** Apache-2.0
- **Homepage:** https://zhuanlan.zhihu.com/p/2031696581678866733

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-shen-shanshan-vllm-dev-skills-vllm-benchmark-result-analysis
- Seller: https://agentstack.voostack.com/s/shen-shanshan
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.