# Run Benchmark

> Run skill quality benchmarks from specs/benchmarks/ definitions and write pass@k reports. Use before and after evolve-skill to prove quality changes are improvements, not regressions.

- **Type:** Skill
- **Install:** `agentstack add skill-danielvm-git-bigpowers-run-benchmark`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [danielvm-git](https://agentstack.voostack.com/s/danielvm-git)
- **Installs:** 0
- **Category:** [Cloud & Infrastructure](https://agentstack.voostack.com/c/cloud-infrastructure)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [danielvm-git](https://github.com/danielvm-git)
- **Source:** https://github.com/danielvm-git/bigpowers/tree/main/.gemini/extensions/bigpowers/skills/run-benchmark
- **Website:** https://github.com/danielvm-git/bigpowers

## Install

```sh
agentstack add skill-danielvm-git-bigpowers-run-benchmark
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Run Benchmark

> **HARD GATE** — Do NOT use benchmark scores to declare a skill "good" or "bad" in isolation. Benchmarks measure relative quality vs. a baseline — they catch regressions, they do not certify correctness.

Reads benchmark definitions from `specs/benchmarks/`, executes each scenario's grader, and writes a structured `pass@k` report that `evolve-skill` consumes.

## Usage

```bash
# Benchmark a single skill
run-benchmark 

# Benchmark all skills with definitions
run-benchmark --all

# Pin current results as baseline
run-benchmark  --baseline
```

## Process

1. **Locate definition** — Read `specs/benchmarks/.yaml`. If absent, report: `"No benchmark definition found for . Create specs/benchmarks/.yaml first."` and stop.

2. **Run each scenario** — For each scenario in `scenarios[]`:
   - **Code grader:** Run `grader.command` in repo root via `bash -c`. Exit 0 → PASS. Non-zero → FAIL. Timeout: 15 seconds.
   - **Rubric grader:** Present each criterion to the agent as a yes/no question about the scenario output. ≥ 80% yes → PASS, else FAIL.

3. **Calculate pass@k** — `pass@k = sum(weight of PASS scenarios) / sum(all weights)`. Round to 2 decimal places.

4. **Write report** to `specs/benchmarks/reports/BENCHMARK--.yaml`:

```yaml
skill: survey-context
run_date: "2026-06-22"
pass_at_k: 0.83
total_scenarios: 3
passed: 2
failed: 1
scenarios:
  - id: s01
    name: "detects active epic from state.yaml"
    result: PASS
    weight: 1.0
  - id: s02
    name: "reads release-plan.yaml and reports next epic"
    result: PASS
    weight: 1.0
  - id: s03
    name: "handles missing state.yaml gracefully"
    result: FAIL
    weight: 0.5
    failure_note: "crashed instead of suggesting state.yaml creation"
```

5. **Baseline mode** (`--baseline`) — Copy the report to `specs/benchmarks/reports/BASELINE-.yaml`. This is the reference point for regression checks in `evolve-skill`.

6. **Compare to baseline** — If a `BASELINE-.yaml` exists, compare `pass_at_k`. Report:
   - `IMPROVED: 0.67 → 0.83`
   - `REGRESSION: 0.83 → 0.67 — do NOT ship this change`
   - `STABLE: 0.83 = 0.83`

## Verify

→ verify: `test -f run-benchmark/SKILL.md && grep -q 'pass_at_k\|pass.at.k' run-benchmark/SKILL.md && echo OK || echo FAIL`

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [danielvm-git](https://github.com/danielvm-git)
- **Source:** [danielvm-git/bigpowers](https://github.com/danielvm-git/bigpowers)
- **License:** MIT
- **Homepage:** https://github.com/danielvm-git/bigpowers

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-danielvm-git-bigpowers-run-benchmark
- Seller: https://agentstack.voostack.com/s/danielvm-git
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.