# Skill Eval Setup

> Auto-generate an evaluation framework (evals.json + test_scenarios.json) for any skill by reading its SKILL.md, then output a ready-to-paste self-improvement loop prompt. Use as "/skill-eval-setup <skill-name-or-path>".

- **Type:** Skill
- **Install:** `agentstack add skill-aberson-claude-skills-skill-eval-setup`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [aberson](https://agentstack.voostack.com/s/aberson)
- **Installs:** 0
- **Category:** [AI & ML](https://agentstack.voostack.com/c/ai-and-ml)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [aberson](https://github.com/aberson)
- **Source:** https://github.com/aberson/claude-skills/tree/main/skill-eval-setup

## Install

```sh
agentstack add skill-aberson-claude-skills-skill-eval-setup
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Skill Eval Setup

Takes a skill name or path, reads its SKILL.md, and produces:
1. An `evals/` folder inside the skill directory
2. `evals/evals.json` — true/false assertions auto-derived from the SKILL.md
3. `evals/test_scenarios.json` — 2–3 synthetic scenarios to test the skill against
4. A copy-paste prompt the user runs in a fresh window to start the self-improvement loop

---

## Input

The user provides one of:
- A skill name (e.g., `session-wrap`) — resolved to `.claude/skills//SKILL.md`
- A relative or absolute path to a skill folder or SKILL.md

Optional flag:
- `--keep-scenarios` — when the target skill already has `evals/test_scenarios.json`, do NOT overwrite it. Regenerate only `evals.json`. Use this when regenerating the assertion set against a rebuilt eval bar while preserving hand-crafted, project-specific scenarios.

If the skill or SKILL.md cannot be found, report the error and stop.

---

## Step 1: Read and analyze the SKILL.md

Read the target SKILL.md completely. Extract every testable requirement:

### What to look for

| Signal | Example | Assertion type |
|---|---|---|
| **Required sections** | "Output two parts: X and Y" | Structure — output contains X and Y |
| **Format rules** | "wrapped in a code fence labeled `text`" | Structure — format matches |
| **Quantity constraints** | "4–8 lines", "2–5 files", "300–600 words" | Structure — count in range |
| **Content requirements** | "must be self-contained", "include the magic word" | Content — specific content present |
| **Specificity rules** | "concrete, not generic", "specific, actionable" | Content — no vague placeholders |
| **Process steps** | "Read MEMORY.md", "Check plan.md" | Process — step was performed |
| **Anti-patterns** | "do NOT duplicate", "skip one-off flukes" | Anti-pattern — bad thing absent |
| **Conditional behavior** | "if X exists, do Y; otherwise do Z" | Conditional — correct branch taken |

### Categorization

Group assertions into categories that match the skill's logical structure. Use the skill's
own section headings as category names when possible. Always include an "Anti-patterns"
category for negative assertions (things the skill must NOT do).

### Assertion format

Each assertion must be:
- A single declarative statement that can be judged TRUE or FALSE
- Testable by reading the skill's output alone (no access to internal state)
- Traceable to a specific line or section in the SKILL.md via the `source` field

---

## Step 1.5: Design discrimination assertions

For each rule extracted in Step 1, name the **defect type** that would violate
the rule. The defect type drives a discrimination assertion: "if the output
exhibits defect D, this assertion grades FALSE."

This is the difference between an eval that catches things and an eval that
just describes things. A shape-checking assertion ("output contains two parts")
passes any well-formed output but cannot tell you whether the assertion is
*useful*. A discrimination assertion ("if either 'Part 1' or 'Part 2' heading
is absent, this assertion grades FALSE") is testable against a deliberately-
broken output corpus.

### Rule → defect type → discrimination assertion

| Rule pattern | Defect type | Discrimination assertion shape |
|---|---|---|
| "Output two parts X and Y" | structural — missing required part | "Output contains both X heading AND Y heading" |
| "wrapped in `text` code fence" | format — wrong fence label | "Output's outer fence label is exactly `text`" |
| "4-8 lines per entry" | quantity-constraint violation | "Every entry has line count in [4, 8]" |
| "must be self-contained" | content — external dependency | "Output references no file path outside the listed Key files set" |
| "do NOT duplicate" | anti-pattern — content-filter | "Output does not contain a memory entry already present in the target memory file" |
| "skip one-off flukes" | anti-pattern — content-filter | "Friction items do not include language matching /typo\|retry\|transient\|attempt/i" |
| "NEVER use triple-backtick fences inside X" | anti-pattern — explicit-rule | "X section contains no occurrences of '```' beyond the outer fence" |
| Fixed-form required bullet ("FIRST bullet must be: ") | required-content — wrong form | "First bullet of `` starts with the literal string ``" |
| Required fixed string in named section ("the magic word for this project is X") | required-content — fixed-string missing | "`` contains the literal string ``" |
| Cross-part claim ("Part 1's `` bullet says YES → Part 2 must not reference prior conversation") | internal-inconsistency — cross-part contradiction | "If Part 1's `` bullet contains ``, Part 2 contains no occurrences of `//`" |
| Conditional behavior ("IF multi-project session, THEN use `/` per-repo separator in Git line") | conditional-behavior — rule not applied | "When `` is present in the output, `` is also present" |

### Pattern derivation guide — patterns most likely to be missed

The example rows above cover structural and anti-pattern rules well. The four
patterns below get systematically missed by template-following and need
explicit derivation.

**1. Fixed-form content.** When the target SKILL.md says "the first X must be
in this exact form" or "must start with the literal string Y", derive an
assertion that checks for the **literal string**, not just that X exists.
Generic "X is present" assertions pass any well-formed X.

**2. Required fixed strings in named sections.** When the SKILL.md names a
specific section ("Required context") AND specifies that a particular literal
string must appear there ("the magic word X"), derive an assertion that
checks for the literal string **inside the named section**.

**3. Cross-part consistency.** When the SKILL.md output has multiple parts
and one part makes a claim about another (e.g., Part 1's `Self-contained:`
bullet asserts whether Part 2 has external dependencies), derive a cross-part
assertion: read the claim in Part 1, then check Part 2 for content that would
contradict it. The assertion's `source:` field must reference BOTH the
part-making-the-claim AND the part-being-claimed-about. Cross-part assertions
are the hardest pattern for template-following to produce — be explicit about
deriving at least one.

**4. Conditional behavior (IF/THEN rules).** Scan the target SKILL.md for
every "If X, …" / "Multi-X sessions: …" / "When X happens, do Y" pattern.
For each, derive an assertion of the form "if `` in
the output, then `` in the output."

### Pair each assertion with a target-skill-section reference

Each assertion's `source:` field must point to the specific lines in the
target SKILL.md that define the violated rule.

**Sourcing precision rule.** For every NEVER / NOT / IF / MUST / "fixed in
form" rule in the target SKILL.md, at least one produced assertion's
`source:` field must name the **specific line** that defines the rule —
either as a standalone number, or as one endpoint of a tight range (≤15
lines wide). A wide line range that brackets but doesn't single out the
rule's line does NOT satisfy this — a reader must be able to locate the
rule from the source string alone.

**Section-heading coverage rule.** Separately from the per-rule precision
above, the produced `source:` fields collectively must reference at least
one line within 10 lines of each major numbered section heading in the
target SKILL.md (Step 1, Step 2, Step 3, etc.). Pure rule-line precision
tends to cluster source citations around sub-rules, leaving section
headings uncovered — derive at least one assertion per major section
whose source field cites the heading line or a line within 10 of it.

### Discrimination quota

Of the assertions in the produced evals.json, **at least 40% must be
discrimination-style** (assertions whose statement names what would make them
grade FALSE, not just what would make them grade TRUE). Shape-only assertions
are still allowed but they are sanity checks, not the primary signal.

Additionally, **at least one of each of the four "patterns most likely to be
missed"** above should appear in the produced evals.json when the target
SKILL.md contains the corresponding rule type. If a pattern type doesn't
apply to a particular target (e.g., target has no multi-part output → no
cross-part consistency assertion possible), note its absence in Part 1
Summary with a one-line "(no X-pattern rules in target — N/A for this skill)".

---

## Step 2: Generate test scenarios

Create 2–3 synthetic test scenarios that exercise different paths through the skill.

### Scenario design principles

- **Cover the main path:** At least one scenario that exercises the skill's primary flow
- **Cover edge cases:** At least one scenario that triggers conditional behavior (e.g., "if
  plan.md is absent", "if the session was clean with no friction")
- **Be self-contained:** Each scenario must include all the context the skill needs to
  produce output — simulated conversation history, file states, user actions
- **Be concrete:** Use realistic project names, file paths, and actions — not "project X"

### Scenario format

```json
{
  "id": "scenario_1",
  "name": "Descriptive name",
  "description": "What this scenario tests",
  "context": "The simulated conversation/environment state the skill sees",
  "expected_assertions": [1, 2, 3, ...]
}
```

The `expected_assertions` field lists which assertion IDs should pass for this scenario.
Typically all assertions apply to all scenarios, but some conditional assertions may only
apply to specific scenarios.

---

## Step 3: Write the eval files

### evals.json

```json
{
  "skill": "",
  "version": "1.0",
  "description": "True/false evaluation for the  skill.",
  "passing_threshold": "/",
  "categories": [
    {
      "name": "Category Name",
      "evals": [
        {
          "id": 1,
          "statement": "The output does X.",
          "source": "SKILL.md, lines NN-MM",
          "defect_type": "",
          "result": null
        }
      ]
    }
  ]
}
```

Rules:
- Set `passing_threshold` to `(total - 2) / total` — allows 2 failures before the skill
  is considered broken
- Number assertions sequentially starting at 1
- The `result` field is always `null` in the template — the loop fills it in
- The `source` field must reference actual line numbers in the SKILL.md (per the
  Sourcing precision and Section-heading coverage rules in Step 1.5)
- The `defect_type` field categorizes the assertion. Use `n/a-sanity` for pure
  structural sanity checks (valid JSON, sequential ids, etc.) and `n/a-coverage`
  for assertions that aggregate over the target SKILL.md rather than testing a
  specific defect.

### test_scenarios.json

```json
{
  "skill": "",
  "version": "1.0",
  "scenarios": [
    {
      "id": "scenario_1",
      "name": "...",
      "description": "...",
      "context": "...",
      "expected_assertions": [1, 2, 3]
    }
  ]
}
```

Write both files to `/evals/`.

### `--keep-scenarios` behavior

When `--keep-scenarios` is passed AND `/evals/test_scenarios.json` already exists:
1. Do NOT overwrite `test_scenarios.json`.
2. Read the existing scenarios and use their `id` values when assigning `expected_assertions` references in the new `evals.json`. If the new assertion set has IDs the old scenarios don't reference (or vice-versa), this is expected — the scenarios stay as-is; only the assertion list is rebuilt.
3. In Part 2 of the output, list `test_scenarios.json` with a `(preserved — N scenarios)` suffix instead of as a written file.

When `--keep-scenarios` is passed BUT no existing `test_scenarios.json` is found, fall through to normal behavior (generate fresh scenarios) and note this in Part 1 Summary.

---

## Step 4: Generate the self-improvement loop prompt

Output the following prompt, customized for the target skill. Present it in a code fence
labeled `text` so the user can copy-paste it into a fresh window.

The prompt must include:
1. The skill name and the **absolute** path to the evals folder (e.g., `.claude/skills//evals/`) — every `` and `` placeholder in the template must be replaced with concrete absolute values
2. Instructions to read the evals.json and test_scenarios.json
3. The produce-then-grade loop: for each scenario, simulate the skill's behavior following
   the SKILL.md instructions, produce the output, then grade every assertion TRUE/FALSE
4. Score calculation: pass count / total assertions across all scenarios
5. If any assertions fail: propose ONE targeted change to SKILL.md, apply it, re-run all
   scenarios, and compare scores
6. Keep/revert logic: keep if score improved, `git reset` if not
7. Iteration logging: number, score, keep/discard, what was tried
8. Termination: perfect score or user interrupt
9. Git commit on each kept change with a message like:
   ` eval: iteration N — `

### Prompt template

```text
Run a self-improvement loop on the skill at ``.

## Setup
- Read `/SKILL.md` (the skill definition)
- Read `/evals/evals.json` (the assertions)
- Read `/evals/test_scenarios.json` (the test scenarios)

## Loop (repeat until perfect score or interrupted)

### Produce
For each scenario in test_scenarios.json:
1. Simulate the environment described in the scenario's `context` field
2. Follow the SKILL.md instructions exactly as if you were executing the skill
3. Produce the full output the skill would generate

### Grade
For each assertion in evals.json, judge the produced output:
- TRUE if the assertion holds for the output
- FALSE if it does not
- When in doubt, grade FALSE (strict grading prevents false passes)

Calculate score: (total TRUE across all scenarios) / (total assertions x number of scenarios)

### Improve
If score /SKILL.md && git commit -m " eval: iteration N — "`
6. If score did not improve: `git checkout -- /SKILL.md`

### Log
Append to `/evals/iteration_log.md` using this format:

    ## Iteration N
    - Score: X/Y (Z%)
    - Result: KEPT / DISCARDED
    - Change: 
    - Failures: 

## Rules
- Do NOT stop to ask questions. Run autonomously.
- Do NOT make more than ONE change per iteration.
- Do NOT modify evals.json or test_scenarios.json.
- If you hit 3 consecutive DISCARDED iterations with no progress, try a fundamentally
  different approach rather than tweaking the same area.
```

Replace all `` and `` placeholders with actual values.

---

## Step 5: Generate the golden bad-example corpus

`/skill-iterate`'s composite scoring (see `_shared/score-skill.md`) requires
a per-skill **golden corpus**: one `good.md` plus one `bad_.md`
per discrimination assertion. The corpus is the gradient signal hill-climbing
optimizes against — without it, shape-only assertions pass 100% on any
well-formed output and the loop has no gradient.

Hand-writing the corpus does not scale (27 evaluable skills × 5–9 defect
types ≈ 130–240 files). This step documents the LLM-driven generation
procedure and the script that automates it.

Run the generator after Step 4 finishes. The generator consumes the
`evals.json` produced by Step 3 and emits the corpus into
`/evals/golden/`.

### Per-skill golden directory layout

```
.claude/skills//evals/golden/
    good.md                       .md        .md
    ...
    manifest.json                 /evals/golden/manifest.json`.

### Slug derivation rule

Map each assertion's `defect_type` field to a filesystem-safe slug:

1. Lowercase the `defect_type`.
2. Replace every run of non-alphanumeric characters with a single `_`.
3. Strip leading and trailing `_`.
4. Truncate at 60 characters.

| `defect_type` | Derived slug | Filename |
|---|---|---|
| `structural — missing required output part` | `structural_missing_required_output_part` | `bad_structural_missing_required_output_part.md` |
| `anti-pattern — explicit-rule violation` | `anti_pattern_explicit_rule_violation` | `bad_anti_pattern_explicit_rule_violation.md` |
| `quantity-constraint violation` | `quantity_constraint_violation` | `bad_quantity_constraint_violation.md` |
| `required-content — fixed-string missing`

…

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [aberson](https://github.com/aberson)
- **Source:** [aberson/claude-skills](https://github.com/aberson/claude-skills)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-aberson-claude-skills-skill-eval-setup
- Seller: https://agentstack.voostack.com/s/aberson
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.