# Skill Forge Benchmark

> >

- **Type:** Skill
- **Install:** `agentstack add skill-agricidaniel-skill-forge-skill-forge-benchmark`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [AgriciDaniel](https://agentstack.voostack.com/s/agricidaniel)
- **Installs:** 0
- **Category:** [Agent Skills](https://agentstack.voostack.com/c/agent-skills)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [AgriciDaniel](https://github.com/AgriciDaniel)
- **Source:** https://github.com/AgriciDaniel/skill-forge/tree/main/skills/skill-forge-benchmark
- **Website:** https://agricidaniel.com

## Install

```sh
agentstack add skill-agricidaniel-skill-forge-skill-forge-benchmark
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Skill Benchmarking & Performance Tracking

Measure and compare skill performance across iterations with statistical
rigor using multiple trials, variance analysis, and trend tracking.

## Process

### Step 1: Define Benchmark Configuration

Accept configuration as:
- **Existing eval set**: Path to `evals/evals.json` (from `/skill-forge eval`)
- **Benchmark config**: Custom config with trial count and thresholds

**Benchmark config schema:**
```json
{
  "skill_name": "my-skill",
  "skill_path": "./my-skill",
  "eval_set_path": "./evals/evals.json",
  "trials_per_eval": 3,
  "baseline_type": "no_skill",
  "previous_benchmark": null,
  "thresholds": {
    "min_pass_rate": 0.8,
    "max_avg_tokens": 100000,
    "max_avg_duration_seconds": 120,
    "min_improvement_ratio": 1.0
  }
}
```

### Step 2: Execute Benchmark Runs

For each eval, run `trials_per_eval` times (default: 3) to get reliable metrics:

1. Execute with-skill runs (3x per eval)
2. Execute baseline runs (3x per eval)
3. Capture per-run: pass/fail, token count, duration
4. Save each run's `timing.json` and `grading.json`

Use `agents/skill-forge-executor.md` for parallel execution where possible.

### Step 3: Aggregate Results

Run `python scripts/aggregate_benchmark.py /iteration- --skill-name `:

**Output `benchmark.json` schema:**
```json
{
  "skill_name": "my-skill",
  "iteration": 1,
  "timestamp": "2026-03-06T12:00:00Z",
  "summary": {
    "total_evals": 10,
    "with_skill": {
      "pass_rate": 0.87,
      "pass_rate_std": 0.05,
      "avg_tokens": 45000,
      "avg_duration_seconds": 34.2
    },
    "baseline": {
      "pass_rate": 0.60,
      "pass_rate_std": 0.08,
      "avg_tokens": 62000,
      "avg_duration_seconds": 52.1
    },
    "improvement_ratio": 1.45,
    "token_savings_ratio": 0.73,
    "time_savings_ratio": 0.66
  },
  "per_eval": [
    {
      "eval_id": 0,
      "eval_name": "basic-trigger",
      "with_skill": {"pass_rate": 1.0, "avg_tokens": 30000, "avg_duration_seconds": 20.1},
      "baseline": {"pass_rate": 0.67, "avg_tokens": 50000, "avg_duration_seconds": 45.0},
      "trials": 3
    }
  ],
  "thresholds_met": {
    "min_pass_rate": true,
    "max_avg_tokens": true,
    "max_avg_duration_seconds": true,
    "min_improvement_ratio": true
  }
}
```

### Step 4: Compare with Previous Iterations

If `previous_benchmark` is provided or prior `iteration-` exists:

1. Load previous `benchmark.json`
2. Calculate delta per metric:
   - Pass rate change
   - Token usage change
   - Duration change
   - New regressions (evals that passed before but fail now)
   - New improvements (evals that failed before but pass now)

### Step 5: Generate Benchmark Report

```markdown
# Benchmark Report: [skill-name]

## Iteration [N] vs [N-1]

### Summary
| Metric | Current | Previous | Delta | Threshold | Status |
|--------|---------|----------|-------|-----------|--------|
| Pass Rate | 87% | 78% | +9% | >= 80% | PASS |
| Avg Tokens | 45K | 52K | -13% | = 1.0x | PASS |

### Regressions (Action Required)
| Eval | Previous | Current | Notes |
|------|----------|---------|-------|
| eval-5 | PASS | FAIL | Output missing required section |

### Improvements
| Eval | Previous | Current | Notes |
|------|----------|---------|-------|
| eval-3 | FAIL | PASS | Error handling now works |

### Per-Eval Detail
[Full breakdown table]

### Variance Analysis
| Eval | Pass Rate | Std Dev | Trials | Reliability |
|------|-----------|---------|--------|-------------|
| eval-0 | 100% | 0.00 | 3 | High |
| eval-1 | 67% | 0.47 | 3 | Low (investigate) |

### Recommendations
[Based on regressions, low-reliability evals, and threshold failures]
```

### Step 6: Threshold Gating

If any threshold fails:
1. Flag as **FAIL** with specific threshold details
2. List which evals caused the failure
3. Recommend running `/skill-forge evolve` to address issues
4. Do NOT approve for publish until thresholds pass

## Error Handling

- **Flaky trials**: If a trial times out or crashes, exclude it from variance calculation and note `"trials_completed"` vs `"trials_requested"` in per-eval results
- **Insufficient trials**: If fewer than 2 trials complete for an eval, flag variance as `"unreliable"` in the report
- **Missing baseline**: If baseline runs fail entirely, report with-skill results only and skip improvement_ratio
- **Threshold edge cases**: If pass_rate equals the threshold exactly, treat as PASS

## Integration with Other Sub-Skills

- **skill-forge-eval**: Provides the eval set and grading infrastructure
- **skill-forge-evolve**: Receives benchmark failures as improvement targets
- **skill-forge-publish**: Requires benchmark pass (score >= thresholds) before publish
- **skill-forge-review**: Can include benchmark summary in review report

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [AgriciDaniel](https://github.com/AgriciDaniel)
- **Source:** [AgriciDaniel/skill-forge](https://github.com/AgriciDaniel/skill-forge)
- **License:** MIT
- **Homepage:** https://agricidaniel.com

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-agricidaniel-skill-forge-skill-forge-benchmark
- Seller: https://agentstack.voostack.com/s/agricidaniel
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.