Skill Eval Setup

Skill Eval Setup

Takes a skill name or path, reads its SKILL.md, and produces:

1. An evals/ folder inside the skill directory
2. evals/evals.json — true/false assertions auto-derived from the SKILL.md
3. evals/test_scenarios.json — 2–3 synthetic scenarios to test the skill against
4. A copy-paste prompt the user runs in a fresh window to start the self-improvement loop

---

Input

The user provides one of:

• A skill name (e.g., session-wrap) — resolved to .claude/skills//SKILL.md
• A relative or absolute path to a skill folder or SKILL.md

Optional flag:

• --keep-scenarios — when the target skill already has evals/test_scenarios.json, do NOT overwrite it. Regenerate only evals.json. Use this when regenerating the assertion set against a rebuilt eval bar while preserving hand-crafted, project-specific scenarios.

If the skill or SKILL.md cannot be found, report the error and stop.

---

Step 1: Read and analyze the SKILL.md

Read the target SKILL.md completely. Extract every testable requirement:

What to look for

| Signal | Example | Assertion type | |---|---|---| | Required sections | "Output two parts: X and Y" | Structure — output contains X and Y | | Format rules | "wrapped in a code fence labeled text" | Structure — format matches | | Quantity constraints | "4–8 lines", "2–5 files", "300–600 words" | Structure — count in range | | Content requirements | "must be self-contained", "include the magic word" | Content — specific content present | | Specificity rules | "concrete, not generic", "specific, actionable" | Content — no vague placeholders | | Process steps | "Read MEMORY.md", "Check plan.md" | Process — step was performed | | Anti-patterns | "do NOT duplicate", "skip one-off flukes" | Anti-pattern — bad thing absent | | Conditional behavior | "if X exists, do Y; otherwise do Z" | Conditional — correct branch taken |

Categorization

Group assertions into categories that match the skill's logical structure. Use the skill's own section headings as category names when possible. Always include an "Anti-patterns" category for negative assertions (things the skill must NOT do).

Assertion format

Each assertion must be:

• A single declarative statement that can be judged TRUE or FALSE
• Testable by reading the skill's output alone (no access to internal state)
• Traceable to a specific line or section in the SKILL.md via the source field

---

Step 1.5: Design discrimination assertions

For each rule extracted in Step 1, name the defect type that would violate the rule. The defect type drives a discrimination assertion: "if the output exhibits defect D, this assertion grades FALSE."

This is the difference between an eval that catches things and an eval that just describes things. A shape-checking assertion ("output contains two parts") passes any well-formed output but cannot tell you whether the assertion is useful. A discrimination assertion ("if either 'Part 1' or 'Part 2' heading is absent, this assertion grades FALSE") is testable against a deliberately- broken output corpus.

Rule → defect type → discrimination assertion

| Rule pattern | Defect type | Discrimination assertion shape | |---|---|---| | "Output two parts X and Y" | structural — missing required part | "Output contains both X heading AND Y heading" | | "wrapped in text code fence" | format — wrong fence label | "Output's outer fence label is exactly text" | | "4-8 lines per entry" | quantity-constraint violation | "Every entry has line count in [4, 8]" | | "must be self-contained" | content — external dependency | "Output references no file path outside the listed Key files set" | | "do NOT duplicate" | anti-pattern — content-filter | "Output does not contain a memory entry already present in the target memory file" | | "skip one-off flukes" | anti-pattern — content-filter | "Friction items do not include language matching /typo\|retry\|transient\|attempt/i" | | "NEVER use triple-backtick fences inside X" | anti-pattern — explicit-rule | "X section contains no occurrences of '``' beyond the outer fence" | | Fixed-form required bullet ("FIRST bullet must be: ") | required-content — wrong form | "First bullet of starts with the literal string " | | Required fixed string in named section ("the magic word for this project is X") | required-content — fixed-string missing | " contains the literal string " | | Cross-part claim ("Part 1's bullet says YES → Part 2 must not reference prior conversation") | internal-inconsistency — cross-part contradiction | "If Part 1's bullet contains , Part 2 contains no occurrences of //" | | Conditional behavior ("IF multi-project session, THEN use / per-repo separator in Git line") | conditional-behavior — rule not applied | "When is present in the output, ` is also present" |

Pattern derivation guide — patterns most likely to be missed

The example rows above cover structural and anti-pattern rules well. The four patterns below get systematically missed by template-following and need explicit derivation.

1. Fixed-form content. When the target SKILL.md says "the first X must be in this exact form" or "must start with the literal string Y", derive an assertion that checks for the literal string, not just that X exists. Generic "X is present" assertions pass any well-formed X.

2. Required fixed strings in named sections. When the SKILL.md names a specific section ("Required context") AND specifies that a particular literal string must appear there ("the magic word X"), derive an assertion that checks for the literal string inside the named section.

3. Cross-part consistency. When the SKILL.md output has multiple parts and one part makes a claim about another (e.g., Part 1's Self-contained: bullet asserts whether Part 2 has external dependencies), derive a cross-part assertion: read the claim in Part 1, then check Part 2 for content that would contradict it. The assertion's source: field must reference BOTH the part-making-the-claim AND the part-being-claimed-about. Cross-part assertions are the hardest pattern for template-following to produce — be explicit about deriving at least one.

4. Conditional behavior (IF/THEN rules). Scan the target SKILL.md for every "If X, …" / "Multi-X sessions: …" / "When X happens, do Y" pattern. For each, derive an assertion of the form "if ` in the output, then ` in the output."

Pair each assertion with a target-skill-section reference

Each assertion's source: field must point to the specific lines in the target SKILL.md that define the violated rule.

Sourcing precision rule. For every NEVER / NOT / IF / MUST / "fixed in form" rule in the target SKILL.md, at least one produced assertion's source: field must name the specific line that defines the rule — either as a standalone number, or as one endpoint of a tight range (≤15 lines wide). A wide line range that brackets but doesn't single out the rule's line does NOT satisfy this — a reader must be able to locate the rule from the source string alone.

Section-heading coverage rule. Separately from the per-rule precision above, the produced source: fields collectively must reference at least one line within 10 lines of each major numbered section heading in the target SKILL.md (Step 1, Step 2, Step 3, etc.). Pure rule-line precision tends to cluster source citations around sub-rules, leaving section headings uncovered — derive at least one assertion per major section whose source field cites the heading line or a line within 10 of it.

Discrimination quota

Of the assertions in the produced evals.json, at least 40% must be discrimination-style (assertions whose statement names what would make them grade FALSE, not just what would make them grade TRUE). Shape-only assertions are still allowed but they are sanity checks, not the primary signal.

Additionally, at least one of each of the four "patterns most likely to be missed" above should appear in the produced evals.json when the target SKILL.md contains the corresponding rule type. If a pattern type doesn't apply to a particular target (e.g., target has no multi-part output → no cross-part consistency assertion possible), note its absence in Part 1 Summary with a one-line "(no X-pattern rules in target — N/A for this skill)".

---

Step 2: Generate test scenarios

Create 2–3 synthetic test scenarios that exercise different paths through the skill.

Scenario design principles

• Cover the main path: At least one scenario that exercises the skill's primary flow
• Cover edge cases: At least one scenario that triggers conditional behavior (e.g., "if

plan.md is absent", "if the session was clean with no friction")

• Be self-contained: Each scenario must include all the context the skill needs to

produce output — simulated conversation history, file states, user actions

• Be concrete: Use realistic project names, file paths, and actions — not "project X"

Scenario format

{
  "id": "scenario_1",
  "name": "Descriptive name",
  "description": "What this scenario tests",
  "context": "The simulated conversation/environment state the skill sees",
  "expected_assertions": [1, 2, 3, ...]
}

The expected_assertions field lists which assertion IDs should pass for this scenario. Typically all assertions apply to all scenarios, but some conditional assertions may only apply to specific scenarios.

---

Step 3: Write the eval files

evals.json

{
  "skill": "",
  "version": "1.0",
  "description": "True/false evaluation for the  skill.",
  "passing_threshold": "/",
  "categories": [
    {
      "name": "Category Name",
      "evals": [
        {
          "id": 1,
          "statement": "The output does X.",
          "source": "SKILL.md, lines NN-MM",
          "defect_type": "",
          "result": null
        }
      ]
    }
  ]
}

Rules:

• Set passing_threshold to (total - 2) / total — allows 2 failures before the skill

is considered broken

• Number assertions sequentially starting at 1
• The result field is always null in the template — the loop fills it in
• The source field must reference actual line numbers in the SKILL.md (per the

Sourcing precision and Section-heading coverage rules in Step 1.5)

• The defect_type field categorizes the assertion. Use n/a-sanity for pure

structural sanity checks (valid JSON, sequential ids, etc.) and n/a-coverage for assertions that aggregate over the target SKILL.md rather than testing a specific defect.

test_scenarios.json

{
  "skill": "",
  "version": "1.0",
  "scenarios": [
    {
      "id": "scenario_1",
      "name": "...",
      "description": "...",
      "context": "...",
      "expected_assertions": [1, 2, 3]
    }
  ]
}

Write both files to /evals/.

--keep-scenarios behavior

When --keep-scenarios is passed AND /evals/test_scenarios.json already exists:

1. Do NOT overwrite test_scenarios.json.
2. Read the existing scenarios and use their id values when assigning expected_assertions references in the new evals.json. If the new assertion set has IDs the old scenarios don't reference (or vice-versa), this is expected — the scenarios stay as-is; only the assertion list is rebuilt.
3. In Part 2 of the output, list test_scenarios.json with a (preserved — N scenarios) suffix instead of as a written file.

When --keep-scenarios is passed BUT no existing test_scenarios.json is found, fall through to normal behavior (generate fresh scenarios) and note this in Part 1 Summary.

---

Step 4: Generate the self-improvement loop prompt

Output the following prompt, customized for the target skill. Present it in a code fence labeled text so the user can copy-paste it into a fresh window.

The prompt must include:

1. The skill name and the absolute path to the evals folder (e.g., .claude/skills//evals/) — every ` and ` placeholder in the template must be replaced with concrete absolute values
2. Instructions to read the evals.json and test_scenarios.json
3. The produce-then-grade loop: for each scenario, simulate the skill's behavior following

the SKILL.md instructions, produce the output, then grade every assertion TRUE/FALSE

1. Score calculation: pass count / total assertions across all scenarios
2. If any assertions fail: propose ONE targeted change to SKILL.md, apply it, re-run all

scenarios, and compare scores

1. Keep/revert logic: keep if score improved, git reset if not
2. Iteration logging: number, score, keep/discard, what was tried
3. Termination: perfect score or user interrupt
4. Git commit on each kept change with a message like:

eval: iteration N —

Prompt template

Run a self-improvement loop on the skill at ``.

## Setup
- Read `/SKILL.md` (the skill definition)
- Read `/evals/evals.json` (the assertions)
- Read `/evals/test_scenarios.json` (the test scenarios)

## Loop (repeat until perfect score or interrupted)

### Produce
For each scenario in test_scenarios.json:
1. Simulate the environment described in the scenario's `context` field
2. Follow the SKILL.md instructions exactly as if you were executing the skill
3. Produce the full output the skill would generate

### Grade
For each assertion in evals.json, judge the produced output:
- TRUE if the assertion holds for the output
- FALSE if it does not
- When in doubt, grade FALSE (strict grading prevents false passes)

Calculate score: (total TRUE across all scenarios) / (total assertions x number of scenarios)

### Improve
If score /SKILL.md && git commit -m " eval: iteration N — "`
6. If score did not improve: `git checkout -- /SKILL.md`

### Log
Append to `/evals/iteration_log.md` using this format:

    ## Iteration N
    - Score: X/Y (Z%)
    - Result: KEPT / DISCARDED
    - Change: 
    - Failures: 

## Rules
- Do NOT stop to ask questions. Run autonomously.
- Do NOT make more than ONE change per iteration.
- Do NOT modify evals.json or test_scenarios.json.
- If you hit 3 consecutive DISCARDED iterations with no progress, try a fundamentally
  different approach rather than tweaking the same area.

Replace all ` and ` placeholders with actual values.

---

Step 5: Generate the golden bad-example corpus

/skill-iterate's composite scoring (see _shared/score-skill.md) requires a per-skill golden corpus: one good.md plus one bad_.md per discrimination assertion. The corpus is the gradient signal hill-climbing optimizes against — without it, shape-only assertions pass 100% on any well-formed output and the loop has no gradient.

Hand-writing the corpus does not scale (27 evaluable skills × 5–9 defect types ≈ 130–240 files). This step documents the LLM-driven generation procedure and the script that automates it.

Run the generator after Step 4 finishes. The generator consumes the evals.json produced by Step 3 and emits the corpus into /evals/golden/.

Per-skill golden directory layout

.claude/skills//evals/golden/
    good.md                       .md        .md
    ...
    manifest.json                 /evals/golden/manifest.json`.

### Slug derivation rule

Map each assertion's `defect_type` field to a filesystem-safe slug:

1. Lowercase the `defect_type`.
2. Replace every run of non-alphanumeric characters with a single `_`.
3. Strip leading and trailing `_`.
4. Truncate at 60 characters.

| `defect_type` | Derived slug | Filename |
|---|---|---|
| `structural — missing required output part` | `structural_missing_required_output_part` | `bad_structural_missing_required_output_part.md` |
| `anti-pattern — explicit-rule violation` | `anti_pattern_explicit_rule_violation` | `bad_anti_pattern_explicit_rule_violation.md` |
| `quantity-constraint violation` | `quantity_constraint_violation` | `bad_quantity_constraint_violation.md` |
| `required-content — fixed-string missing`

…

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [aberson](https://github.com/aberson)
- **Source:** [aberson/claude-skills](https://github.com/aberson/claude-skills)
- **License:** MIT

Install and usage instructions live in the source repository linked above.