Skill Forge

Skill Forge

Create, judge, and improve skills. Every rule in this skill traces to evidence from a corpus study of 1400+ public skills, deep-analyzed in waves (anthropics/skills, obra/superpowers, openai/skills, Vercel/Cloudflare/Supabase/Expo/HF/Microsoft/Trail of Bits, Matt Pocock, Addy Osmani, baoyu, marketing/PM verticals, K-Dense). Rule IDs like U7/S8/R15 and the spike exemplars in references/exemplars.md trace to that study (methodology summarized in the project README). The skill works standalone; the evidence is provenance, not a runtime dependency.

A skill is not a document. It is three layers of machinery: a corrective patch for the model's bad defaults (not a restatement of domain knowledge), an optimal allocation of knowledge across context budget (progressive disclosure), and a falsifiable quality loop (tests that can fail). Drop any layer and you produce one of the twenty documented anti-patterns.

Mode routing

| User intent sounds like | Mode | Read next | |---|---|---| | "turn this into a skill", "create a skill for X", 做成技能 | A: Create | below + references/type-playbooks.md | | "is this skill good?", "audit/score this skill", 评估这个skill | B: Judge | references/rubric.md | | "improve/fix/refresh this skill", 优化这个skill | C: Improve | references/rubric.md then below | | "the skill doesn't trigger / triggers too much" | D: Tune description | references/description-engineering.md |

Output language for user-facing reports: Chinese. Skill bodies you produce: English by default (instructions read by models), bilingual trigger terms in the description when the user works in Chinese; follow the user's explicit preference if stated.

Mode A: Create

Eight steps. Steps 1-2 are gates — they can kill the project, which is a success, not a failure. Do not skip gates to be agreeable; a skill that shouldn't exist costs context on every future session.

Step 1 — Supply gate (R1)

A skill must be built from real, observed expertise. Acceptable sources:

• (a) A task actually completed in this or a recorded conversation — extract the steps that worked, the corrections the user made, the input/output formats observed.
• (b) A named existing asset: runbook, incident report, review comments, a personal knowledge base you can search (Obsidian vault, notes directory, wiki), or an upstream skill to adapt.
• (c) Documented external truth you can fetch and cite (official docs, changelogs, spec).

If none exists, say so and offer two honest paths: do the task once for real and harvest it, or produce a thin draft with explicit OPEN QUESTIONS the user must answer. Never fabricate procedures from general knowledge — the corpus shows this produces "vague, generic procedures" (the #1 supply-side failure; U9).

Step 2 — Deletion test (R2)

Ask: if this skill did not exist, would the model acting on its priors get it wrong?

• Wrong → full skill justified.
• Mostly right → build the thin version only: version pitfalls + safety lines + ecosystem routing, a few dozen lines. (Evidence: scikit-learn-style API restatements fail this test; pydeseq2-style trap collections pass it.)
• Right → recommend no skill; a one-line note in an existing skill or rules file may suffice.

Also apply per-section while drafting: "delete this paragraph — does model behavior change?" If not, cut it.

Step 3 — Type and neighbors (R3, R4)

Classify into one of six types; the type selects the structure template, freedom defaults, and test mechanism. Read references/type-playbooks.md for the chosen type before drafting.

| Type | One-line test | |---|---| | technique | Changes how the model works on any task (discipline, style) | | workflow | Multi-step process with stages, gates, handoffs | | reference | Knowledge the model must consult (rules, APIs, conventions) | | tool-wrapper | Teaches a CLI/API/script the model wouldn't know | | domain | Domain judgment, taste, conventions, compliance | | meta | Operates on skills/agents/rules themselves |

Then scan for neighbors: list your installed skill directories (~/.agents/skills/, ~/.claude/skills/, and any project .agents/skills/) plus a skill registry/index if you keep one. If any overlap exists, the new skill MUST contain a disambiguation line ("X is handled here; for Y use Z") and the neighbor should gain a reciprocal pointer. Overlapping skills without mutual routing silently steal each other's triggers — observed even inside official repos.

Step 4 — Description engineering (R5)

The description is the skill's only activation surface — write it before the body, against references/description-engineering.md. Core rules: three parts (what-scope + "Use when" triggers + NOT-boundary/routing); trigger terms ranked by specificity (error strings, proper nouns, file extensions, verbatim user phrases > generic verbs, max 3 generic verbs); bilingual triggers for Chinese-language usage; never summarize the workflow's steps in the description (verified failure: the model follows the summary and skips the body); no "Always use ... even if" unless it is the domain's only entry point.

Step 5 — Draft the body (R7-R14)

Budget first: target 100-350 lines, hard ceiling 500; total knowledge >1500 lines forces a references/ split (one level deep only; >10k-word files need a grep-anchor index). Critical constraints go in the first screen, never after line 1000.

While drafting, enforce the type playbook plus these cross-type rules:

• Anti-default inventory (R10): list the model's known bad defaults in this domain (aesthetic, language, behavior, stale knowledge) and write each as a checkable countermeasure — a ban table, numeric threshold, or "X not Y" override. This is the highest-value content a skill can carry (U7).
• Freedom per subtask (R8): fragile/sequential subtasks get scripts or verbatim templates (low freedom); judgment subtasks get principles + anti-pattern fences + false-positive lists (high freedom). Mix freely within one skill.
• Output contract (R12): deliverable-producing skills define the artifact's format (field table, fixed sections, badge, machine-parseable line) on the first screen.
• Run-time verification (R16): build "assume there are problems" checks into the skill itself — a fix-and-verify loop, an until-clean command, a quality gate on the artifact. Generation is not completion.
• Tables over prose (R13) for decisions, routing, and error handling; prose only to explain why a rule exists (explained rules generalize; bare MUSTs don't).
• Negative boundary (R14): every skill gets "When NOT to use" with an alternative. Style/persistent skills also get an exit phrase and a safety exception.
• Freshness protocol (R11): any version/price/limit/model-name gets one of: checked stamp + refresh path + "docs win" conflict rule; deliberate omission + official link; or a regenerate command.
• Discipline three-piece (R9): technique/discipline skills require a Rationalization table (excuse → reality), Red Flags list, and a letter-equals-spirit clause — sourced from observed excuses (Step 6 RED run), not invented ones.
• Danger guards (R21): skills that scan, write, publish, or attack get triple authorization statements, disable-model-invocation: true where user-initiated-only, and narrow allowed-tools. Prefer harness enforcement (hooks, allowed-tools) over polite instructions (R22): what can be mechanically blocked must not rely on model goodwill.

Step 6 — Test (R15, R16, R17)

Route by type; protocols and minimal local variants in references/testing-protocols.md:

| Type | Primary | Minimum acceptable | |---|---|---| | technique | Pressure test (RED baseline, record verbatim excuses → countermeasures) | 1 no-skill pressure scenario + verify the skill blocks the observed excuse | | workflow | Eval loop (with-skill vs baseline subagents) | 2 real prompts, both arms, human compare | | reference | Deletion test + reference lint + trigger eval | Lint + 3 trigger probes | | tool-wrapper | Golden tests (sample input → expected output shape) | 1 golden test executed | | domain | Behavior assertions (process/format, not content) + human review | evals.json with 2 cases | | meta | Eval loop + self-compliance lint (it obeys its own rules) | Lint passes on itself |

Store test prompts in evals/evals.json (schema: skill_name, evals[].id/prompt/expected_output/assertions). Evals are development assets: they must exist and be kept; whether they ship in a package is a secondary choice. When improving an existing skill, snapshot it first — the old version is your baseline.

Step 7 — Lint (R18-R20)

Run the validator; it must pass with zero errors before delivery:

python3 ~/.agents/skills/skill-forge/scripts/validate_skill.py  [--json] [--strict]

It checks: frontmatter field whitelist and name/description spec rules, reference existence (every references/, scripts/, assets/ path mentioned must exist — dangling references are the highest-severity documented defect), line budgets, imperative-inflation density (MANDATORY/⚠️/CAPS), keyword-stuffing signals, secret patterns in examples, and promised-vs-actual file counts. The validator is also the Mode B pre-pass.

Step 8 — Install and register

1. Install to ~/.agents/skills// for a user-level skill, or /.agents/skills// for a project skill. (~/.agents/skills/ is the cross-tool convention read natively by Codex and other agents.)
2. Claude Code native discovery: symlink ln -s ~/.agents/skills/ ~/.claude/skills/ (verify it resolves).
3. If your setup maintains a skill registry/index/catalog, rebuild it and add a route row (trigger, input, output, path, priority) so the new skill is discoverable and de-duplicated against neighbors.
4. Restart the agent session so skill discovery refreshes.

Mode B: Judge

Read references/rubric.md and score the target against the 14 signals in causal order — S8 deletion test first (should it exist?), then S1 triggering (can it activate?), then S6/S7 structure and noise (can it execute?), then S10/S11 verification and contract (can its output be trusted?). A failure upstream makes downstream scores moot. Run the validator (Step 7) as the mechanical pre-pass; never hand-check what the script checks.

Default judging tier is validator + manual reasoning against the anchors. Signals whose detection column says subagent (S8 baseline run, S1 trigger probes) may be scored manually at this tier — state in the report that the probe was not executed; escalate to real probe runs only when the verdict hinges on them or the user asks for a deep audit.

Deliver: per-signal score with quoted evidence, the 2-3 highest-leverage fixes mapped to playbook patterns, and an honest verdict line: keep / improve / rebuild thin / retire. Do not pad findings to seem thorough — 3 real issues beat 10 cosmetic ones (the-fool's rule: cap challenges, no nitpick-stacking).

Mode C: Improve

1. Judge first (Mode B) — fixes must trace to scored failures, not taste.
2. Snapshot the current version (cp -r to a workspace) as the eval baseline.
3. Apply fixes per the relevant playbook section; re-run lint and the type's test mechanism against the snapshot.
4. Improvement philosophy (evidence-backed): generalize from feedback instead of overfitting to the test examples; cut what isn't pulling its weight (read transcripts — wasted model actions point at cuttable text); explain why instead of stacking MUSTs; if several test runs each hand-wrote a similar helper, that helper becomes scripts/.

Rationalizations to reject (this skill's own discipline)

| Excuse | Reality | |---|---| | "The request is clear, skip the supply gate" | Clear requests still produce generic slop without observed expertise; the gate takes one question | | "It's a small skill, no tests needed" | The minimum tier exists precisely for small skills; zero tests = "generation is the finish line", the documented top failure mode | | "Lint later, deliver first" | Dangling references are written exactly at delivery time; lint costs seconds | | "User is in a hurry, skip the deletion test" | A skill that shouldn't exist taxes every future session's context — slower forever to save a minute now | | "I'll write the description from the body summary" | Verified failure mode: models execute the description's summary and skip the body |

Red flags you are about to violate this skill: writing the body before the description; a description containing step words ("first analyzes, then..."); delivering with "the skill is ready" without a lint run in the same turn; inventing an excuse table instead of running a RED observation.

When NOT to use

• Routing/catalog questions about installed skills → agent-skill-hub.
• Lark API skill content → lark-skill-maker for the domain template; come back here for description, tests, and lint.
• One-off prompts, CLAUDE.md/AGENTS.md rules, subagent definitions → not skills; write them directly.
• In Claude Code with the user explicitly asking for Anthropic's interactive eval-viewer flow → anthropic-skills:skill-creator plugin remains available; its run_loop.py (description optimization) and eval-viewer can be reused from here for Step 6/D since they only require claude -p.

Resource map

| File | Load when | |---|---| | references/rubric.md | Mode B/C; final QA of Mode A | | references/type-playbooks.md | Mode A Step 3-5; Mode C fixes | | references/description-engineering.md | Step 4; Mode D | | references/testing-protocols.md | Step 6; Mode C regression | | references/anti-patterns.md | Drafting QA; judging evidence lookup | | references/exemplars.md | Mode A Step 5 — imitate the verified best exemplar of the chosen type (52 spike exemplars, citation-verified) | | scripts/validate_skill.py | Step 7; Mode B pre-pass (execute, don't read) | | scripts/verify_citations.py | Maintaining evidence-backed assets — quotes must exist verbatim (execute) | | scripts/build_exemplars.py | Regenerate exemplars.md from verified JSON (execute) |

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: truthandstories07-sudo
• Source: truthandstories07-sudo/skill-forge
• License: MIT

Install and usage instructions live in the source repository linked above.