Skill Evaluator

Skill Evaluator

Audit a target skill to see whether its instructions actually land when Claude runs it. This skill does NOT benchmark skill-vs-no-skill (that's skill-creator's job). This skill checks rule adherence under realistic prompts, classifies where failures come from, and proposes targeted fixes.

Scope

Good fit — v1 supports:

• Workflow skills (sequences of steps, shutdown rituals, review passes)
• Rule-shaped skills (policies, conventions, constraints, checklists)
• Guideline skills (style guides, voice guides, design rules)

Out of scope for v1 (say so explicitly if asked):

• Creative-synthesis skills (brand voice, canvas design, algorithmic art) — rule-adherence framing does not apply cleanly to "is this output beautiful"
• Skills whose value is measured by end-user outcomes over time (engagement, retention)

If the user asks to evaluate a creative-synthesis skill, say the harness is not validated for that skill type and ask if they want to proceed anyway with a best-effort run. Do not silently pretend it works.

Differentiation from skill-creator

| skill-creator | skill-evaluator | |--|--| | Authors new skills | Audits existing skills | | Benchmarks skill-vs-baseline (A/B) | Measures rule adherence under realistic prompts | | Asks "does the skill help?" | Asks "does the skill's text actually land?" | | Outputs: a skill | Outputs: findings + rule-text diffs |

You can chain them: evaluator finds a gap → creator's conventions guide the rule-text fix.

Harness lens — what to audit beyond rule adherence

Rule-adherence audits catch whether the agent follows what the skill says. The harness lens catches a different class of failure: whether the skill is shaped right in the first place. Run these eight questions in addition to (not instead of) the standard adherence checks. They're cheap; one or two will usually surface a real issue.

1. Does the skill name its primitives? A well-shaped skill is explicit about which of the five harness primitives it serves: context engineering, progressive disclosure, observable feedback loops, state preservation, eval discipline. If the skill mixes all five implicitly, it probably has a scope problem. Flag: "Skill doesn't name what kind of work it's doing for the agent."
2. Progressive disclosure or front-loaded? Is the SKILL.md body trying to be the encyclopedia, or is it a map that points to references/ for detail? If body length > ~300 lines without clear section-then-reference structure, the skill is front-loading context that should be lazy. Flag: "Body should be a table of contents; details belong in references/."
3. Are feedback loops machine-checkable, or only prose? Look for rules of the form "the agent should consider X" with no audit, no reviewer, no checkable artifact. Those rules drift. Flag: "Rule X is aspirational — propose a structured check (linter / reviewer subagent / explicit artifact gate)."
4. Environment-failure vs. prompt-failure misdiagnosis. When the skill describes a known failure mode and prescribes "try harder" prompting, the diagnosis is probably wrong. Ask: what capability is missing from the agent's environment? (a tool, a reference file, an upstream artifact, an explicit gate). Flag: "Rule Y treats an environment failure as a prompting failure — consider adding [specific capability]."
5. State-preservation gap. Does the skill produce artifacts a future session can pick up, or does it dump output to chat? For workflow skills that span sessions, output going only to chat is a state-preservation failure. Flag: "Outputs should land at a predictable path so a follow-up session can resume without re-briefing."
6. Undefined-state coverage. A skill's happy path quietly depends on things existing — an upstream artifact, a canonical file, a populated field, a prior step's output. For each such dependency ask: does the skill define what happens when it's absent? If not, that is an undefined state — the skill will improvise there, usually badly, and a rule-adherence audit will not catch it because there is no rule to adhere to or violate. Flag: "Rule X assumes [resource] exists but defines no behavior when it's missing — add an explicit absent-state branch." Pairs with the absent-state test category in Phase 3: this question names the gap, Phase 3 tests it.
7. Is every word load-bearing? Beyond whether rules land, check how tightly they're phrased. Two failure shapes: (a) prose bloat — sentences that restate, hedge, or pad without changing what the agent does; (b) vague phrasing — soft verbs and adjectives ("handle appropriately", "be thorough", "make it good") where a precise operation would remove the guesswork. Flag: "Rule X pads/hedges — cut to the load-bearing clause" or "Rule X uses a vague verb ('handle') — name the precise operation." Caveat: economy serves clarity, not brevity for its own sake — a why line that prevents a misread earns its tokens, so do not flag it. (Pattern adapted from nidhinjs/prompt-master, MIT.)
8. Cross-platform loadability. Claude imposes no description length limit, but OpenAI Codex silently skips any skill whose frontmatter description exceeds 1024 UTF-8 bytes (or trips its name/angle-bracket rules) — so a skill can pass every adherence check above and still be invisible on Codex. Don't eyeball it. In this repo, run python3 scripts/check-skill-compat.py and report any FAIL; for a skill elsewhere, measure the description's UTF-8 byte length (not char count — —/→/CJK cost 2–4 bytes each) against 1024. Flag: "description is N bytes (>1024) — Codex skips this skill; trim to ≤1024 B keeping triggers and disambiguation boundaries (see CLAUDE.md's cross-platform frontmatter contract), and push overflow into instructions/body." This is a portability gate, not a behavior finding — fix it before the behavioral audit, since an unloadable skill has no behavior to audit on that platform.

Treat each finding the same way as adherence findings: classify by fix layer (skill text / rubric / brief / fixture) and propose targeted diffs. The harness lens does not change the rest of the workflow.

Dependencies

• skill-creator (Anthropic-shipped) — recommended. Used to enforce authoring conventions when you propose rule-text diffs. Suggest installation if the user doesn't have it.
• User-authored orchestration skills (e.g., sub-agent-coordinator) — not required. Opt-in only.

Artifact policy

Default: dry run. The evaluator prints to chat and writes no files. This keeps the target skill's folder clean and the user's workspace untouched — especially important when the target skill lives inside the mounted workspace.

Hard rules (no exceptions):

1. Never write inside the target skill's folder. Not during the run, not for saving findings, not ever.
2. Do not create directories at the workspace root by default. No auto-created _evaluation/ or similar folders. The workspace is the user's, not ours.
3. Intermediate state lives in Claude's session sandbox only — scratch space outside the user's workspace, cleared when the session ends. The user does not need to see or inspect it.
4. Save only on explicit request. If the user says "save the findings", "keep this report", or similar, write ONE file at the workspace root with a clear name: skill-evaluation-{skill-name}-{YYYY-MM-DD}.md. Never a folder. Never multiple files. Never inside the skill.
5. Confirm the path before writing. Example: "Saving to skill-evaluation-cerby-2026-04-17.md at your workspace root — confirm?"

These rules apply even when the user explicitly asks to override them. If the user says "save it inside the skill folder" or "create an _eval/ subfolder there", decline and offer the workspace-root alternative. Do not present an "override per your explicit request" option. The rules exist because the harness must not contaminate the audited skill's git history; user intent does not change that constraint.

If the user also wants full traces (executor output, grader output per test), treat it as a separate opt-in. Same filename convention: skill-evaluation-{skill-name}-{YYYY-MM-DD}-traces.md. Still one file, still at the workspace root.

If no workspace is mounted: tell the user, offer to keep the report inline so they can copy-paste it. Do not guess a save location.

Workflow — 7 Phases

Run these phases in order. Do not skip Phase 2 even if the target skill looks obvious.

Phase 0 — Boundary check (before Phase 1)

Before reading the target skill, check the incoming request for two patterns this skill does not handle on its own:

1. Build-and-audit mixing — phrases like "build me a skill and audit it", "create and then review", "scaffold then test", or any request that combines authoring a new skill with auditing it. This skill audits existing skills. Hand the build step to skill-creator first and resume here only after a SKILL.md exists on disk. Never draft the SKILL.md inline as part of fulfilling an audit request.
2. A/B benchmarking — phrases like "benchmark this skill", "skill-vs-no-skill", "measure quality lift", "does the skill help". This skill measures rule adherence, not output quality. Decline and point to skill-creator's run_eval primitive (see "When to refuse").

If neither pattern applies, proceed to Phase 1. If one applies, take the boundary action (chain to skill-creator or refuse) before touching the target skill.

Why this matters: the triggering metadata in the frontmatter description already declares these boundaries, but metadata controls triggering, not behavior-after-triggering. Once this skill is loaded, the workflow body is what executors follow. Phase 0 puts the boundary check inside the body where it will actually fire.

Phase 1 — Read the target skill

Read the target SKILL.md in full. Then read every file it references (via references/, assets/, scripts/, or inline links). A skill's rules often live in reference files, not in SKILL.md itself — evaluating SKILL.md alone will miss 30-60% of the surface area.

Build a mental index:

• Purpose (from the description field and opening paragraph)
• Trigger phrases (from the description field)
• Rules / steps / constraints (the testable content)
• Explicit non-goals or "out of scope" sections

Why this matters: if you generate test prompts without reading the reference files, your assertions will miss rules that only live there. The evaluation will show false passes.

Phase 2 — Clarify (≤3 questions, only if needed)

Ask the user targeted questions ONLY if the skill text does not make the following obvious:

1. Purpose ambiguity — what problem does this skill solve?
2. Trigger ambiguity — which kinds of requests should invoke it?
3. Success criteria — what does "working correctly" look like?

Budget: 3 questions max. If the skill text is clear on all three, skip this phase entirely.

Why the budget: fully autonomous evaluation compounds bad inference into bad tests → bad findings. Three questions is usually enough to avoid garbage-in-garbage-out without turning this into a workshop.

If the user gives a one-word skill name with no context and you cannot figure out the three items above from the skill text alone, ask all three at once (not sequentially).

Phase 3 — Generate test prompts + assertions

Produce 6–12 test prompts that span the skill's declared surface area:

• Happy path (2–3) — requests the skill's description directly matches
• Edge cases (2–3) — requests at the boundary of the trigger phrases
• Adjacent non-matches (1–2) — requests that look like they should trigger the skill but shouldn't, to check for over-triggering
• Rule-specific stress tests (1–2) — requests designed to make Claude violate a specific rule in the skill
• Absent-state tests (1–2) — for every resource a rule quietly assumes exists (an upstream artifact, a canonical file, a populated field, a prior step's output), write a request where that resource is absent. These catch the states a skill's happy path depends on but never defines behavior for — the most common source of bugs that slip an audit. Load-bearing: the fixture must actually withhold the resource. A fixture that supplies it "to be realistic" hides the exact bug — the absence is the fixture.

For each test prompt, write 3–7 assertions using the tag/sentence/evidence pattern (see references/assertion-dictionary.md). Every assertion must be independently gradable from the executor's output alone — no need for the grader to re-read the skill to judge it.

Calibration for load-bearing assertions (web-output skills). When the target skill produces rendered web output (HTML, SVG, DOM) AND an assertion would change a release decision if it failed, recommend a calibrated grader pair: a gold-standard fixture the grader MUST pass, and a negative fixture deliberately broken on the rule's dimension that the grader MUST fail. This catches vacuous assertions (grader passes on both) and over-strict assertions (grader fails on both). Pattern in references/calibration-loop.md; concrete Playwright shape in references/playwright-grader-shape.md. Pure workflow skills have no rendered surface to grade — skip this step for them.

Output format:

test_N.md
---
## Prompt

## Assertions
- [T1] The executor calls the shutdown ritual. Evidence: mention of "shutdown" or explicit QA sub-agent spawn.
- [T2] ...

Phase 4 — Split-role evaluation

> Run this skill in the main loop — it must be able to spawn sub-agents. Phase 4 > is the bias removal: it spawns fresh-context executor + grader sub-agents. If > this skill is itself running nested (you were dispatched as a sub-agent) or the > platform has no sub-agent dispatch, you cannot spawn them — and a simulated > "split" in one context is not independent grading. Do NOT silently simulate. > In that case, switch to the Degraded mode in the Platform Fallback section > below: emit the DEGRADED banner, do the best in-context pass you can, and label > every finding as non-independent. Never present simulated grading as if the > split-role harness ran.

Run each test via two sub-agents with fresh context:

Executor sub-agent

• Receives: the test prompt + the target skill loaded
• Mode: stated-intent by default (describe tool calls with -stated suffix, don't actually execute). Switch to live-execute only if the skill under test is purely advisory (no side effects).
• Output: a single response showing reasoning, tool-call intents, and final deliverable

Grader sub-agent

• Receives: the test prompt, the executor's output, and the assertion list
• Does NOT receive the target skill's text (to avoid bias toward what the skill says over what the executor did)
• Output: per-assertion pass/fail with one-line evidence quote
• Fresh context per test (mandatory). Each test gets its own grader sub-agent invocation. Do not batch tests into a shared grader. See references/grader-brief.md "Fresh context per test" for the rule and rationale.

Why split roles: having the same agent execute and grade introduces bias. Fresh grader context with only the assertion list forces evidence-based judgment. See references/executor-brief.md and references/grader-brief.md for the briefs.

High-stakes mode (optional, opt-in): for regulated audits, safety-critical ship gates, or when the user explicitly asks to double-check grading, run a second grader in fresh context on the same inputs. Disagreements demote to unclear and surface in a "Disputed assertions" subsection of the findings report. Cost roughly doubles, so default is off. Triggers, quorum rules, and reporting are in references/grader-brief.md "High-stakes mode — optional second-grader quorum".

Phase 5 — Classify

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: sorawit-w
• Source: sorawit-w/agent-skills
• License: MIT

Install and usage instructions live in the source repository linked above.